Image enhancement method and electronic device

Abstract

The present application provides an image enhancement method and an electronic device for realizing image enhancement, and solving the problem that the image quality algorithm existing in the prior art does not meet the real-time requirements of video processing on the mobile terminal. The method includes: the electronic device obtains a color image in the YUV color space, and the CPU of the electronic device enlarges the image size of the first U channel component and the first V channel component of the color image to obtain the processed second U channel component and the second V channel component, and the NPU of the electronic device performs super-resolution processing on the first Y channel component of the color image to obtain the processed second Y channel component; in addition, the NPU of the electronic device performs high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the three channel components respectively, and finally, the CPU of the electronic device synthesizes the enhanced color image according to the component coefficients corresponding to the three channels and the processed channel components.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image enhancement method and electronic equipment. Background Art

[0002] With the popularity of smart phones, the number of mobile video users is increasing rapidly, and users' requirements for picture quality are also increasing. Therefore, it is necessary to be able to adaptively adjust parameters such as brightness and contrast in the video to optimize and enhance the image quality.

[0003] Image enhancement is to improve the quality and visual effect of images by using a series of image enhancement technologies, highlight the features of interest in the image, obtain valuable information in the image, and convert the image into a form that is more suitable for human or machine analysis and processing, so that the processed image has better effects for certain specific applications. Image enhancement theory is widely used in biomedical fields, industrial production fields, public security fields, and aerospace fields.

[0004] In the current related technologies, one image enhancement method is to use a super resolution (SR) technology to upconvert the input image into a high-resolution output image. Another image enhancement method is to use high dynamic range imaging (HDR) technology to expand the value of each pixel in an input image with a standard dynamic range to a high dynamic range. For smartphones, the decoding results of the video decoder on the smartphone side are usually in the YUV color space, not the RGB color space, and the current image enhancement methods are all based on the video in the RGB color space for model training and inference generation, which is not suitable for use on mobile platforms such as smartphones. Moreover, the current image enhancement methods are based on super-resolution algorithms based on deep learning and machine learning, which have a large amount of calculation and cannot meet the real-time requirements of mobile terminal video processing, and the power consumption is also large. Summary of the invention

[0005] The present application provides an image enhancement method and an electronic device to achieve a better image enhancement effect and solve the problem that the image quality algorithm in the prior art does not meet the real-time requirements of mobile terminal video processing.

[0006] In a first aspect, an embodiment of the present application provides an image enhancement method, including: an electronic device acquires a color image in a YUV color space, the electronic device enlarges the image size of a first U channel component and a first V channel component of the color image to obtain a processed second U channel component and a second V channel component, and performs super-resolution processing on the first Y channel component of the color image to obtain a processed second Y channel component; in addition, the electronic device performs high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively.

[0007] Further, the electronic device obtains an enhanced channel component corresponding to the first Y channel component based on the component coefficients corresponding to the second Y channel component and the first Y channel component, obtains an enhanced channel component corresponding to the first U channel component based on the component coefficients corresponding to the second U channel component and the first U channel component, and obtains an enhanced channel component corresponding to the first V channel component based on the component coefficients corresponding to the second V channel component and the first V channel component. Finally, the electronic device synthesizes an enhanced color image based on the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component, and the enhanced channel component corresponding to the first V channel component.

[0008] In the embodiment of the present application, the electronic device adopts different image enhancement methods for different channels of the color image in the YUV color space, performs super-resolution processing on the color image in the YUV color space, and performs high dynamic range imaging processing, thereby improving the image enhancement effect.

[0009] In one possible design, the CPU of the electronic device amplifies the first U channel component and the first V channel component of the color image to obtain the amplified second U channel component and the second V channel component. At the same time, the NPU of the electronic device performs super-resolution processing on the first Y channel component of the color image to obtain the processed second Y channel component; and the NPU of the electronic device performs high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the three channels. It can be seen that the CPU and NPU of the electronic device execute in parallel, which can improve the image processing speed to meet the real-time requirements of mobile terminal video processing.

[0010] In a possible design, the enhanced channel component corresponding to the first Y channel component may satisfy: Y'=W Y ×Y; where Y' represents the enhanced channel component corresponding to the first Y channel component, W Y represents the component coefficient corresponding to the first Y channel component, and Y represents the second Y channel component;

[0011] The enhanced channel component corresponding to the first U channel component can satisfy: Among them, U' represents the enhanced channel component corresponding to the first U channel component, W U represents the component coefficient corresponding to the first U channel component, U represents the second U channel component, is a fixed parameter, is a fixed parameter, and The value range of is [0,1];

[0012] The enhanced channel component corresponding to the first V channel component can satisfy: Where V' represents the enhanced channel component corresponding to the first V channel component, W V represents the component coefficient corresponding to the first V channel component, V represents the second V channel component, is a fixed parameter, is a fixed parameter, and The value range is [0,1].

[0013] It can be seen that in the embodiment of the present application, the electronic device not only performs super-resolution processing on the color image in the YUV color space, but also performs high dynamic range imaging processing, which can improve the image enhancement effect.

[0014] In a possible design, the component coefficient corresponding to the U channel is the same as the component coefficient corresponding to the V channel, which can avoid color deviation.

[0015] In a possible design, the NPU of the electronic device performs super-resolution processing on the first Y channel component input into the dual-branch multi-task neural network model, and outputs the second Y channel component; at the same time, the NPU of the electronic device performs high dynamic range imaging processing on the color image input into the dual-branch multi-task neural network model, and outputs component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively. That is, the NPU can simultaneously complete super-resolution enhancement and high dynamic range imaging enhancement of the video.

[0016] In one possible design, the electronic device includes a video decoder; the video decoder is used to decode an input video stream to generate a color image, wherein each frame of the input video stream is a color image in an RGB color space. That is, the image enhancement method provided in the embodiment of the present application can implement the above-mentioned image enhancement processing on the image decoded by the decoder.

[0017] In one possible design, the electronic device includes a video encoder. After the electronic device synthesizes the enhanced color image, the video encoder can also encode each frame of the enhanced color image to generate a video stream after image enhancement processing, and each frame of the image in the video stream after image enhancement processing is a color image in the RGB color space. In other words, the image enhancement method provided in the embodiment of the present application can implement encoder encoding of the image in the YUV color space after image enhancement, and after the encoder encoding, it is an image in the RGB color space.

[0018] In a second aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory is used to store one or more computer programs; when the one or more computer programs stored in the memory are executed by the processor, the electronic device is able to implement any possible design method of any of the above aspects.

[0019] In a third aspect, the embodiment of the present application further provides a device, which includes a module / unit for executing any possible design method of any of the above aspects. These modules / units can be implemented by hardware, or by executing corresponding software implementations by hardware.

[0020] In a fourth aspect, a computer-readable storage medium is also provided in an embodiment of the present application, and the computer-readable storage medium includes a computer program. When the computer program runs on an electronic device, the electronic device executes any possible design method of any of the above aspects.

[0021] In a fifth aspect, an embodiment of the present application further provides a method comprising a computer program product, which, when executed on a terminal, enables the electronic device to execute any possible design of any of the above aspects.

[0022] In a sixth aspect, an embodiment of the present application further provides a chip, which is coupled to a memory and is used to execute a computer program stored in the memory to execute any possible design method of any of the above aspects.

[0023] For the technical effects that can be achieved by the various designs in any of the second to sixth aspects above, please refer to the description of the technical effects that can be achieved by the various designs in the first aspect above, and no further details will be given here. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of the structure of a mobile phone provided in an embodiment of the present application;

[0025] Figure 2 A schematic diagram of the structure of an Android operating system provided in an embodiment of the present application;

[0026] Figure 3 A schematic diagram of an electronic device is provided for an embodiment of the present application;

[0027] Figure 4 A schematic diagram of a dual-branch multi-task neural network model is provided for an embodiment of the present application;

[0028] Figure 5A and Figure 5B A schematic diagram of an image enhancement method is provided for an embodiment of the present application;

[0029] Figure 6 An exemplary schematic diagram of an image enhancement method is provided for an embodiment of the present application;

[0030] Figure 7 A schematic diagram of an image enhancement device is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0031] In order to facilitate understanding of the embodiments of the present application, the following first introduces terms related to the embodiments of the present application.

[0032] 1. Color space. Color is usually described by three independent attributes. The three independent variables work together to form a space coordinate, which is the color space. But the color object being described is objective. Different color spaces just measure the same object from different angles. Color space can be divided into two categories: the first category is the primary color space, such as the red, green, blue (RGB) color space; the second category is the color and brightness separation color space, such as the YUV color space, where "Y" represents brightness (luminance or luma), which is the grayscale value; and "U" and "V" represent chrominance (chrominance or chroma), and hue, saturation, value (HSV) color space, etc.

[0033] 2. Components. The three components of RGB color space can be red, green, and blue; the three components of YUV color space can be YUV components. Among them, "Y" represents brightness (luminance or luma), that is, grayscale value; and "U" and "V" represent chrominance (chroma).

[0034] For mobile terminals, after obtaining the video stream, the mobile terminal needs to decode it through a video decoder first. The color image obtained by decoding is usually a color image in the YUV color space, rather than a color image in the RGB color space. In the YUV color space, the Y channel reflects the brightness of the image in the video stream, while the UV channel reflects the color of the image in the video stream. Generally speaking, the human eye is more sensitive to brightness and less sensitive to subtle differences in color. This requires that when processing the video stream in the YUV color space, different image enhancement methods should be adopted for different channels. Based on this analysis, an embodiment of the present application provides an image enhancement method, which can realize super-resolution processing of the video stream in the YUV color space, high dynamic range enhancement processing of the video stream, and synthesize the output image after super-resolution processing and the output image after high dynamic range enhancement processing to generate a final enhanced image, thereby better improving the quality of the image displayed by the mobile terminal.

[0035] Among them, the method and the device are based on the same technical concept. Since the principles of solving problems by the method and the device are similar, the implementation of the device and the method can refer to each other, and the repeated parts will not be repeated.

[0036] The present application can be applied to video streaming playback scenarios, such as mobile phone video playback, video playback in video applications, or playback of television multimedia content. The image enhancement method provided in the present application can be applied to electronic devices including those with a central processing unit (CPU) and an embedded neural network processor (NPU), including but not limited to cameras, video cameras, smart cameras, smart camcorders, smart mobile terminals (such as mobile phones, tablets, etc.), smart TVs and other portable terminals. The image enhancement method provided in the present application can also be applied to the video playback device in the video surveillance system, or implemented by the cloud server in the video surveillance system. Exemplary embodiments of portable terminals include but are not limited to those equipped with Or a portable terminal with other operating systems. The portable terminal may also be a laptop computer with a touch-sensitive surface (e.g., a touch panel). It should also be understood that in some other embodiments, the terminal may also be a desktop computer with a touch-sensitive surface (e.g., a touch panel).

[0037] The following takes the electronic device as a mobile phone as an example. Figure 1 A schematic structural diagram of the mobile phone 100 is shown.

[0038] The mobile phone 100 may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 151, a wireless communication module 152, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a SIM card interface 195, etc. The sensor module 180 may include a gyroscope sensor 180A, an acceleration sensor 180B, a pressure sensor 180C, a proximity light sensor 180G, a fingerprint sensor 180H, and a touch sensor 180K (of course, the mobile phone 100 may also include other sensors, such as a temperature sensor, a distance sensor, a magnetic sensor, an ambient light sensor, an air pressure sensor, a bone conduction sensor, etc., which are not shown in the figure).

[0039] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the mobile phone 100. In other embodiments of the present application, the mobile phone 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0040] The processor 110 may include one or more processing units, for example: the processor 110 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural network processor (Neural-network Processing Unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. Among them, the controller can be the nerve center and command center of the mobile phone 100. The controller can generate an operation control signal according to the instruction opcode and the timing signal to complete the control of fetching and executing instructions.

[0041] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0042] The processor 110 can run the image enhancement method provided in the embodiment of the present application to achieve better image enhancement effects and solve the problem that the image quality algorithm in the prior art does not meet the real-time requirements of mobile terminal video processing. The processor 110 can include different devices. For example, when a CPU and a GPU are integrated, the CPU and the GPU can cooperate to execute the image enhancement method provided in the embodiment of the present application. For example, part of the algorithm in the image enhancement method is executed by the CPU, and another part of the algorithm is executed by the GPU to obtain faster processing efficiency.

[0043] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the mobile phone 100 may include 1 or N display screens 194, where N is a positive integer greater than 1. In the embodiment of the present application, the touch sensor and / or pressure sensor on the display screen 194 can collect the user's touch operation, and the touch sensor and / or pressure sensor can transmit the detected sensor data to the processor 110 so that the processor 110 determines the corresponding state of the sensor unit.

[0044] In the embodiment of the present application, the display screen 194 may be an integrated flexible display screen, or a spliced ​​display screen consisting of two rigid screens and a flexible screen located between the two rigid screens. After the processor 110 runs the image enhancement method provided in the embodiment of the present application, the processor 110 may control the image enhancement effect of the display interface on the display screen 194 based on the image enhancement method.

[0045] The camera 193 (a front camera or a rear camera, or one camera can be used as both a front camera and a rear camera) is used to capture still images or videos. Generally, the camera 193 may include a photosensitive element such as a lens group and an image sensor, wherein the lens group includes a plurality of lenses (convex lenses or concave lenses) for collecting light signals reflected by the object to be photographed and transmitting the collected light signals to the image sensor. The image sensor generates an original image of the object to be photographed based on the light signal.

[0046] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the mobile phone 100 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store codes of operating systems, applications (such as camera applications, WeChat applications, etc.), etc. The data storage area can store data created during the use of the mobile phone 100 (such as data collected by sensors, and a preset reference state sequence set), etc.

[0047] The internal memory 121 may also store the code of the image enhancement algorithm provided in the embodiment of the present application. When the code of the image enhancement algorithm stored in the internal memory 121 is executed by the processor 110, the processor 110 may control the display interface on the display screen 194.

[0048] In addition, the internal memory 121 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0049] Of course, the code of the image enhancement algorithm provided in the embodiment of the present application can also be stored in an external memory. In this case, the processor 110 can run the code of the image enhancement algorithm stored in the external memory through the external memory interface 120, and the processor 110 determines the enhanced image of the electronic device, and then controls the display interface on the display screen 194 according to the enhanced image.

[0050] The functions of the sensor module 180 are described below.

[0051] The gyroscope sensor 180A can be used to determine the motion posture of the mobile phone 100. In some embodiments, the angular velocity of the mobile phone 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180A. That is, the gyroscope sensor 180A can be used to detect the current motion state of the mobile phone 100, such as shaking or stillness, such as horizontal screen or vertical screen.

[0052] When the display screen in the embodiment of the present application is a foldable screen, the gyroscope sensor 180A can be used to detect a folding or unfolding operation on the display screen 194. The gyroscope sensor 180A can report the detected folding operation or unfolding operation as an event to the processor 110 to determine the folded state or unfolded state of the display screen 194.

[0053] The acceleration sensor 180B can detect the magnitude of the acceleration of the mobile phone 100 in various directions (generally three axes). When the display screen in the embodiment of the present application is a foldable screen, the acceleration sensor 180B can be used to detect the folding or unfolding operation acting on the display screen 194. The acceleration sensor 180B can report the detected folding operation or unfolding operation as an event to the processor 110 to determine the folded state or unfolded state of the display screen 194.

[0054] The pressure sensor 180C is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180B can be arranged on the display screen 194 or the shell part. There are many types of pressure sensors 180B, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180B, the capacitance between the electrodes changes. The mobile phone 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the mobile phone 100 detects the touch operation intensity according to the pressure sensor 180B. The mobile phone 100 can also calculate the touch position according to the detection signal of the pressure sensor 180B. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity greater than the first pressure threshold acts on both sides of the shell, an instruction to view unread messages is executed.

[0055] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The mobile phone emits infrared light outward through the light emitting diode. The mobile phone uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the mobile phone. When insufficient reflected light is detected, the mobile phone can determine that there is no object near the mobile phone. When the display screen in the embodiment of the present application is a non-foldable screen, the proximity light sensor 180G may be set on the upper side of the screen of the display screen 194, and the proximity light sensor 180G may detect whether a face is close to the screen based on the optical path difference of the infrared signal. When the display screen in the embodiment of the present application is a foldable screen, the proximity light sensor 180G may be set on the first screen of the foldable display screen 194, and the proximity light sensor 180G may detect the size of the folding angle or unfolding angle between the first screen and the second screen based on the optical path difference of the infrared signal.

[0056] The fingerprint sensor 180H is used to collect fingerprints. The mobile phone 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0057] The touch sensor 180K is also called a "touch panel". The touch sensor 180K can be arranged on the display screen 194 or the shell part, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can transmit the detected sensor data to the processor 110, so that the processor 110 determines the state of the sensor unit according to the sensor data, and then determines the state sequence corresponding to the sensor unit of the electronic device. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be arranged on the surface of the mobile phone 100, which is different from the position of the display screen 194.

[0058] Exemplarily, the display screen 194 of the mobile phone 100 displays a main interface, which includes icons of multiple applications (such as a camera application, a WeChat application, etc.). The user clicks the icon of the camera application in the main interface through the touch sensor 180K, triggering the processor 110 to start the camera application and turn on the camera 193. The display screen 194 displays the interface of the camera application, such as a viewfinder interface.

[0059] The wireless communication function of the mobile phone 100 can be realized through antenna 1, antenna 1, mobile communication module 151, wireless communication module 152, modem processor and baseband processor.

[0060] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in mobile phone 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0061] The mobile communication module 151 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the mobile phone 100. The mobile communication module 151 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 151 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 151 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 151 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 151 can be set in the same device as at least some of the modules of the processor 110.

[0062] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 151 or other functional modules.

[0063] The wireless communication module 152 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the mobile phone 100. The wireless communication module 152 can be one or more devices integrating at least one communication processing module. The wireless communication module 152 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering processing, and sends the processed signal to the processor 110. The wireless communication module 152 can also receive the signal to be sent from the processor 110, modulate the frequency of the signal, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0064] In addition, the mobile phone 100 can implement audio functions through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone interface 170D, and the application processor. For example, music playback, recording, etc. The mobile phone 100 can receive input from the key 190 and generate key signal input related to the user settings and function control of the mobile phone 100. The mobile phone 100 can use the motor 191 to generate a vibration prompt (such as a vibration prompt for an incoming call). The indicator 192 in the mobile phone 100 can be an indicator light, which can be used to indicate the charging status, power changes, and can also be used to indicate messages, missed calls, notifications, etc. The SIM card interface 195 in the mobile phone 100 is used to connect the SIM card. The SIM card can be inserted into the SIM card interface 195, or pulled out from the SIM card interface 195, to achieve contact and separation with the mobile phone 100.

[0065] It should be understood that in actual applications, the mobile phone 100 may include Figure 1 The embodiments of the present application are not limited to the more or fewer components shown. The illustrated mobile phone 100 is only an example, and the mobile phone 100 may have more or fewer components than those shown in the figure, may combine two or more components, or may have different component configurations. The various components shown in the figure may be implemented in hardware, software, or a combination of hardware and software including one or more signal processing and / or application specific integrated circuits.

[0066] The software system of the electronic device may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to exemplify the software structure of the electronic device. Figure 2 It is a software structure block diagram of the electronic device according to an embodiment of the present invention.

[0067] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system library, and the kernel layer.

[0068] The application layer can include a series of application packages.

[0069] like Figure 2 As shown, the application package may include applications such as phone, camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0070] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0071] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0072] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0073] Content providers are used to store and retrieve data and make it accessible to applications. The data may include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0074] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0075] The phone manager is used to provide communication functions for electronic devices, such as the management of call status (including answering, hanging up, etc.).

[0076] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0077] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.

[0078] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0079] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.

[0080] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0081] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0082] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0083] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0084] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0085] A 2D graphics engine is a drawing engine for 2D drawings.

[0086] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, and sensor driver. Among them, hardware can refer to various sensors, such as the acceleration sensor, gyroscope sensor, touch sensor, pressure sensor, etc. involved in the embodiments of the present application.

[0087] See also Figure 3 As shown, the electronic device involved in the embodiment of the present application may include at least one processor 301, a memory 302, a video encoder 304, and a video decoder 303. The processor 301 may include a CPU 301A and an NPU 301B. The processor 301 may refer to Figure 1 The processor 110 in the memory 302 may refer to Figure 1 The internal memory 121 in.

[0088] Combination Figure 3 Generally speaking, the image enhancement method provided by the present application includes the following processes: 1) first, the video decoder 303 decodes the input video stream in the RGB color space to obtain one or more frames of input images in the YUV color space; 2) then the one or more frames of input images in the YUV color space are transmitted to the CPU 301A for data preprocessing; 3) the image processed by the CPU 301A is transmitted to the dual-branch multi-task neural network model in the NPU 301B for processing; 4) after being processed by the neural network model, a super-resolution (SR) image after super-resolution processing and component coefficients after high dynamic range imaging processing are obtained, and the SR image and component coefficients are transmitted back to the CPU 301A to execute the post-processing thread to synthesize the final enhanced image; 5) the enhanced image is transmitted to the video encoder 304 through the NPU 301B, and is encoded into a video stream in the RGB color space by the video encoder 304 and then output.

[0089] Among them, the dual-branch multi-task neural network model in NPU301B is generated in advance through training. The dual-branch multi-task neural network model includes two branches. The first branch corresponds to the super-resolution processing task, and the second branch corresponds to the high dynamic range imaging processing task. The super-resolution processing task and the high dynamic range imaging processing task can be executed in parallel.

[0090] like Figure 4As shown, the first branch of the dual-branch multi-task neural network model can include 4 convolutional layers and 4 relu (activation) layers, and the first branch is used to perform super-resolution processing on the input color image. Among them, the fourth convolutional layer is a sub-pixel convolutional layer, which means that the output of this layer will be upsampled to the target resolution. The first convolutional layer 401a in the first branch expands the feature dimension, for example, from 3 to 32, and the following second convolutional layer 401b and third convolutional layer 401c extract local features of each pixel on the feature map. The fourth sub-pixel convolutional layer 401d is used to upsample the feature map and enlarge it to the target resolution. The output result of the fourth sub-pixel convolutional layer 401d will be activated by the relu (activation) layer to ensure that the value of each pixel is greater than zero.

[0091] In addition, the second branch of the dual-branch multi-task neural network model may include 4 convolutional layers and 4 relu (activation) layers. The second branch is used to perform high dynamic range imaging on the input color image. Among them, the fourth convolutional layer is a sub-pixel convolutional layer, and the second branch trains the network to predict the weight mask map instead of directly predicting the value of each pixel.

[0092] In addition, the electronic device may also include one or more of the following: a general-purpose processor, an image signal processor (ISP), a microprocessor, a digital signal processor (DSP), a field-programmable gate array (FPGA), etc.

[0093] The memory 302 in the electronic device can be a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM) or other types of dynamic storage devices that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by the device, but is not limited thereto. The memory can be independent and connected to the processor 301 through a bus (such as Figure 3The memory may also be integrated with the processor 301.

[0094] Among them, the memory 302 can be used to store application code for executing the solution of the present application, and the execution is controlled by the processor 301. That is, the processor is used to execute the program code stored in the memory 302 to implement the image enhancement method in the embodiment of the present application.

[0095] The image enhancement method provided in the embodiment of the present application is described in detail below. The image enhancement method can be implemented by an image enhancement device in an electronic device. The image enhancement device can be integrated into the program code of the electronic device as a plug-in. For specific methods, see Figure 5A and Figure 5B shown.

[0096] Step 501: The electronic device obtains a color image in a YUV color space.

[0097] The color image includes a first Y channel component, a first U channel component and a first V channel component.

[0098] In a possible example, the input image of the electronic device is a color image in the RGB color space, and the video decoder of the electronic device needs to decode the input video stream to generate the color image. For example, when the video application of the electronic device receives a play request from the user, the electronic device obtains the video stream from the video server corresponding to the video application, and each frame of the color image in the video stream is a color image in the RGB color space. Figure 5B As shown, the input color image is a color image in the RGB color space, and the video decoder of the electronic device decodes the video stream from the network side, and each frame of the color image after decoding is a color image in the YUV color space.

[0099] Step 502: The electronic device enlarges the image size of the first U channel component and the first V channel component to obtain processed second U channel component and second V channel component, and performs super-resolution processing on the first Y channel component to obtain processed second Y channel component.

[0100] In this step, the electronic device can simultaneously scale the first U channel color component and the first V channel color component in the decoded color image in the YUV color space by a bilinear interpolation algorithm. Figure 5B For example, the CPU of the electronic device can simultaneously amplify the first U channel color component and the first V channel color component in the decoded color image through a bilinear interpolation algorithm to obtain an amplified second U channel component and a second V channel component.

[0101] Considering that the human eye is more sensitive to changes in brightness than to changes in color, in the embodiment of the present application, during the image enhancement process, the electronic device performs super-resolution processing on the first Y channel component separately, and combines Figure 5B For example, the dual-branch multi-task neural network model in the NPU of the electronic device performs super-resolution processing on the Y channel component of the input color image separately to achieve the purpose of targeted optimization of the image based on the characteristics of the YUV color space.

[0102] Step 503: The electronic device performs high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the first Y channel component, the first U channel component and the first V channel component respectively.

[0103] It should be noted that the component coefficient in the embodiment of the present application is also called a weight mask map, and the value range of the component coefficient can be from 0 to 1. The component coefficient under each channel represents the change coefficient of each pixel value under the Y channel and the U channel and the V channel respectively. The so-called change coefficient of each pixel value refers to calculating the difference between the target component of the pixel and the component of the current image for the pixel of any channel, and then taking the ratio between the absolute value of the difference and the target component as the change coefficient of the pixel value, that is, as the component coefficient.

[0104] Combination Figure 5B For example, the dual-branch multi-task neural network model in the NPU of the electronic device performs high dynamic range imaging processing on the input color image to obtain component coefficients corresponding to the first Y channel component, component coefficients corresponding to the first U channel component, and component coefficients corresponding to the first V channel component.

[0105] It should be noted that, in a possible embodiment, for the same frame of color image, the above step 502 and step 503 can be executed in parallel, that is, the CPU of the electronic device amplifies the first U channel component and the first V channel component to obtain the amplified second U channel component and the second V channel component; the dual-branch multi-task neural network model in the NPU of the electronic device performs super-resolution processing on the second Y channel component of the input color image, and the dual-branch multi-task neural network model in the NPU of the electronic device performs high dynamic range imaging processing on the input color image to obtain the component coefficient corresponding to the first Y channel component, the component coefficient corresponding to the first U channel component, and the component coefficient corresponding to the first V channel component. In this way, the image processing efficiency can be further improved to meet the real-time requirements of the mobile terminal for video processing.

[0106] Based on the fact that the first branch and the second branch in the dual-branch multi-task neural network model in the NPU have the ability to process in parallel, the dual-branch multi-task neural network model in the NPU of the electronic device performs super-resolution processing on the first Y channel component of the input color image. At the same time, the dual-branch multi-task neural network model in the NPU of the electronic device performs high dynamic range imaging processing on the input color image to obtain the component coefficient corresponding to the first Y channel component, the component coefficient corresponding to the first U channel component, and the component coefficient corresponding to the first V channel component. In this way, the image processing efficiency can be further improved on the basis of the parallel processing of the CPU and the NPU to meet the real-time requirements of the mobile terminal for video processing.

[0107] Step 504: The electronic device obtains an enhanced rear channel component corresponding to the first Y channel component based on the component coefficients corresponding to the second Y channel component and the first Y channel component, obtains an enhanced rear channel component corresponding to the first U channel component based on the component coefficients corresponding to the second U channel component and the first U channel component, and obtains an enhanced rear channel component corresponding to the first V channel component based on the component coefficients corresponding to the second V channel component and the first V channel component.

[0108] Specifically, for the Y channel, the electronic device can multiply the component coefficient corresponding to the first Y channel component and the second Y channel component to obtain the enhanced channel component corresponding to the first Y channel component. That is, the electronic device obtains the color value of the Y channel by combining the weight mask map of the Y channel predicted by the dual-branch multi-task neural network model in the NPU with the Y channel upsampling result of the pixel value output by the dual-branch multi-task neural network model. Figure 5B For example, the enhanced channel component corresponding to the first Y channel component satisfies: Y'=W Y ×Y; wherein Y' represents the enhanced channel component corresponding to the first Y channel component, W Y represents the component coefficient corresponding to the first Y channel component, and Y represents the second Y channel component.

[0109] In addition, for the U channel, the enhanced channel component corresponding to the first U channel component satisfies: Wherein, U' represents the enhanced channel component corresponding to the first U channel component, W U represents the component coefficient corresponding to the first U channel component, U represents the second U channel component, is a fixed parameter, is a fixed parameter, and The value range is [0,1].

[0110] In addition, for the V channel, the enhanced channel component corresponding to the first V channel component satisfies: Wherein, V' represents the enhanced channel component corresponding to the first V channel component, W V represents the component coefficient corresponding to the first V channel component, V represents the second V channel component, is a fixed parameter, is a fixed parameter, and The value range is [0,1].

[0111] In the embodiment of the present application, in order to avoid color deviation between the component after the U channel enhancement and the component after the V channel enhancement, W can be set in the embodiment of the present application. U and W V The values ​​are the same.

[0112] Step 505: The electronic device synthesizes an enhanced color image according to the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component, and the enhanced channel component corresponding to the first V channel.

[0113] That is, the electronic device calculates the component coefficients predicted by the NPU and the components of each channel after super-resolution processing to synthesize the enhanced color image in the YUV color space.

[0114] In a possible example, the video encoder in the electronic device can also encode the enhanced color images of each frame to obtain a video stream after image enhancement processing, and the various color images in the video stream after image enhancement processing are color images in the RGB color space. Exemplarily, when the video application of the electronic device receives a play request from the user, the electronic device can obtain the video stream from the video server corresponding to the video application, and the electronic device performs the above-mentioned image enhancement processing on the video stream, and then encodes it through the video encoder, and the color image in the RGB color space finally output can be displayed on the display.

[0115] The enhancement method provided in the embodiment of the present application can use the CPU and GPU in parallel to improve the image processing efficiency. Figure 6 As shown. For the CPU of the electronic device, the embodiment of the present application sets two threads, pre-processing and post-processing, and for the NPU of the electronic device, the embodiment of the present application sets a neural network inference thread. These three threads process data in parallel, thereby accelerating the calculation. Figure 6For example, for the i-th color image obtained after decoding the video stream, the pre-processing thread of the CPU of the electronic device executes: performing the amplification process as shown in step 502 on the i-th color image, and then the neural network inference thread of the NPU of the electronic device executes: performing the super-resolution process as shown in step 502 and the high dynamic range imaging process as shown in step 503 on the i-th color image. After that, the post-processing thread of the CPU of the electronic device executes: performing the calculation process as shown in step 504 on the i-th color image, and performing the process of synthesizing the enhanced image as shown in step 505. Figure 6 As shown, during the NPU executing the above processing, the pre-processing thread of the CPU can simultaneously execute: performing the amplification processing as shown in step 502 on the i+1 frame color image, and during the electronic CPU post-processing thread executing the above processing, the neural network inference thread of the NPU of the electronic device can simultaneously execute: performing the super-resolution processing as shown in step 502 and the high dynamic range imaging processing as shown in step 503 on the i+1 frame color image. By analogy, the CPU and NPU parallel computing is realized, so the present invention can perform real-time video processing on the processor (including the CPU and the NPU) of the mobile terminal, thereby improving the image quality of the video without occupying more network bandwidth.

[0116] Based on the same technical concept as the above method, the present application also provides an image enhancement device, see Figure 7 As shown, the image enhancement device may include an image acquisition unit 701, which is used to acquire a color image in a YUV color space; a first processing unit 702, which is used to scale the first U channel component and the first V channel component to obtain an enlarged second U channel component and a second V channel component. A second processing unit 703, which is used to perform super-resolution processing on the first Y channel component to obtain a processed second Y channel component; and perform high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the first Y channel component, the first U channel component and the first V channel component respectively; the first processing unit 702 is further used to obtain an enhanced channel component corresponding to the first Y channel component according to the component coefficients corresponding to the second Y channel component and the first Y channel component, and obtain an enhanced channel component corresponding to the first U channel component according to the component coefficients corresponding to the second U channel component and the first U channel component, and obtain an enhanced channel component corresponding to the first V channel component according to the component coefficients corresponding to the second V channel component and the first V channel component; the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component and the enhanced channel component corresponding to the first V channel component are synthesized.

[0117] In a possible embodiment, the enhanced channel component corresponding to the first Y channel component satisfies: Y'=W Y ×Y; wherein Y' represents the enhanced channel component corresponding to the first Y channel component, W Y represents the component coefficient corresponding to the first Y channel component, and Y represents the second Y channel component;

[0118] The enhanced channel component corresponding to the first U channel component satisfies: Wherein, U' represents the enhanced channel component corresponding to the first U channel component, W U represents the component coefficient corresponding to the first U channel component, U represents the second U channel component, is a fixed parameter, is a fixed parameter, and The value range of is [0,1];

[0119] The enhanced channel component corresponding to the first V channel component satisfies: Wherein, V' represents the enhanced channel component corresponding to the first V channel component, W V represents the component coefficient corresponding to the first V channel component, V represents the second V channel component, is a fixed parameter, is a fixed parameter, and The value range is [0,1].

[0120] In a possible embodiment, the component coefficient corresponding to the first U channel component is the same as the component coefficient corresponding to the first V channel.

[0121] In a possible embodiment, the device further includes a decoding unit 704, which is specifically used to decode the input video stream to generate a color image in the YUV color space, wherein each frame of the image in the input video stream is a color image in the RGB color space.

[0122] In a possible embodiment, the device also includes an encoding unit 705, which is specifically used to encode each frame of the enhanced color image after the first processing unit 702 synthesizes the enhanced color image, so as to generate an image-enhanced video stream, wherein each frame of the image in the image-enhanced video stream is a color image in the RGB color space.

[0123] It should be noted that the division of units in the embodiments of the present application is schematic and is only a logical function division. There may be other division methods in actual implementation. Each functional unit in the embodiments of the present application may be integrated into a processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0124] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0125] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0126] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0127] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0128] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. An image enhancement method, characterized in that: include: The electronic device acquires a color image, wherein the color image includes a first Y channel component, a first U channel component, and a first V channel component; The electronic device enlarges the image size of the first U channel component and the first V channel component to obtain a processed second U channel component and a second V channel component, and performs super-resolution processing on the first Y channel component to obtain a processed second Y channel component; The electronic device performs high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively; the component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively, are expressed as variation coefficients of each pixel value in the color image under the Y channel, the U channel, and the V channel, respectively; The electronic device obtains an enhanced channel component corresponding to the first Y channel component according to the component coefficients corresponding to the second Y channel component and the first Y channel component, obtains an enhanced channel component corresponding to the first U channel component according to the component coefficients corresponding to the second U channel component and the first U channel component, and obtains an enhanced channel component corresponding to the first V channel component according to the component coefficients corresponding to the second V channel component and the first V channel component; The electronic device synthesizes an enhanced color image according to the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component, and the enhanced channel component corresponding to the first V channel component.

2. The method according to claim 1, characterized in that The enhanced channel component corresponding to the first Y channel component satisfies: ;in, represents the enhanced channel component corresponding to the first Y channel component, represents the component coefficient corresponding to the first Y channel component, Represents the second Y channel component; The enhanced channel component corresponding to the first U channel component satisfies: ;in, represents the enhanced channel component corresponding to the first U channel component, represents the component coefficient corresponding to the first U channel component, represents the second U channel component, is a fixed parameter, is a fixed parameter, and The value range of is [0,1]; The enhanced channel component corresponding to the first V channel component satisfies: ;in, represents the enhanced channel component corresponding to the first V channel component, represents the component coefficient corresponding to the first V channel component, represents the second V channel component, is a fixed parameter, is a fixed parameter, and The value range is [0,1].

3. The method according to claim 1 or 2, characterized in that: The component coefficient corresponding to the first U channel component is the same as the component coefficient corresponding to the first V channel.

4. The method according to claim 1, characterized in that: The electronic device comprises an embedded neural network processor NPU and a central processing unit CPU, wherein the NPU is integrated with a dual-branch multi-task neural network model; The NPU of the electronic device performs super-resolution processing on the first Y channel component input into the dual-branch multi-task neural network model, and outputs the second Y channel component, and the CPU of the electronic device performs image size enlargement on the first U channel component and the first V channel component to obtain processed second U channel component and second V channel component; The NPU of the electronic device performs high dynamic range imaging processing on the color image input into the dual-branch multi-task neural network model, and outputs component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively.

5. The method according to claim 1, characterized in that The electronic device acquires a color image, including: The video decoder of the electronic device decodes the input video stream to generate the color image, wherein each frame of the image in the input video stream is a color image in the RGB color space.

6. The method according to claim 1, characterized in that After the electronic device synthesizes the enhanced color image, the electronic device further comprises: The video encoder of the electronic device encodes each frame of the enhanced color image to generate a video stream after image enhancement processing, and each frame of the image in the video stream after image enhancement processing is a color image in the RGB color space.

7. An electronic device, characterized in that: The electronic device comprises a processor and a memory; The memory stores program instructions; The processor is used to run the program instructions stored in the memory, so that the electronic device executes: Acquire a color image, wherein the color image includes a first Y channel component, a first U channel component, and a first V channel component; Performing image size enlargement on the first U channel component and the first V channel component to obtain processed second U channel component and second V channel component, and performing super-resolution processing on the first Y channel component to obtain processed second Y channel component; Performing high dynamic range imaging processing on the color image to obtain component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively; the component coefficients corresponding to the first Y channel component, the first U channel component, and the first V channel component, respectively, are expressed as variation coefficients of each pixel value in the color image under the Y channel, the U channel, and the V channel, respectively; Obtain an enhanced channel component corresponding to the first Y channel component according to the component coefficients corresponding to the second Y channel component and the first Y channel component, obtain an enhanced channel component corresponding to the first U channel component according to the component coefficients corresponding to the second U channel component and the first U channel component, and obtain an enhanced channel component corresponding to the first V channel component according to the component coefficients corresponding to the second V channel component and the first V channel component; An enhanced color image is synthesized according to the enhanced channel component corresponding to the first Y channel component, the enhanced channel component corresponding to the first U channel component, and the enhanced channel component corresponding to the first V channel component.

8. The electronic device according to claim 7, characterized in that: The enhanced channel component corresponding to the first Y channel component satisfies: ;in, represents the enhanced channel component corresponding to the first Y channel component, represents the component coefficient corresponding to the first Y channel component, Represents the second Y channel component; The enhanced channel component corresponding to the first U channel component satisfies: ;in, represents the enhanced channel component corresponding to the first U channel component, represents the component coefficient corresponding to the first U channel component, represents the second U channel component, is a fixed parameter, is a fixed parameter, and The value range of is [0,1]; The enhanced channel component corresponding to the first V channel component satisfies: ;in, represents the enhanced channel component corresponding to the first V channel component, represents the component coefficient corresponding to the first V channel component, represents the second V channel component, is a fixed parameter, is a fixed parameter, and The value range is [0,1].

9. The electronic device according to claim 7 or 8, characterized in that: The component coefficient corresponding to the first U channel component is the same as the component coefficient corresponding to the first V channel component.

10. The electronic device according to claim 7, characterized in that: The processor includes an embedded neural network processor NPU and a central processing unit CPU, and the NPU is integrated with a dual-branch multi-task neural network model; The NPU is configured to run the program instructions stored in the memory, so that the NPU specifically executes: Performing super-resolution processing on the first Y channel component input into the dual-branch multi-task neural network model, and outputting the second Y channel component; The CPU is configured to run the program instructions stored in the memory, so that the CPU specifically executes: Enlarging the image size of the first U channel component and the first V channel component to obtain processed second U channel component and second V channel component; The NPU is also used to run the program instructions stored in the memory, perform high dynamic range imaging processing on the color image input into the dual-branch multi-task neural network model, and output component coefficients corresponding to the first Y channel component, the first U channel component and the first V channel component, respectively.

11. The electronic device according to claim 7, characterized in that: The electronic device further comprises a video decoder; The video decoder is used to decode the input video stream, generate the color image and send it to the processor, wherein each frame of the image in the input video stream is a color image in the RGB color space.

12. The electronic device according to claim 7, characterized in that: The electronic device also includes a video encoder; The video encoder is used to encode each frame of the enhanced color image to generate a video stream after image enhancement processing, and each frame of the image in the video stream after image enhancement processing is a color image in the RGB color space.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or an instruction, and when the program or the instruction is read and executed by one or more processors, the image enhancement method according to any one of claims 1 to 6 can be implemented.

Images