Image enhancement method and image enhancement apparatus

Abstract

The application discloses an image enhancement method and device in the field of computer vision in the field of artificial intelligence, and the image enhancement method comprises the following steps: acquiring a first high dynamic range (HDR) image corresponding to a to-be-processed image and color image features of the to-be-processed image, the to-be-processed image is an image with a first resolution, the first HDR image is an image with a second resolution, the first resolution is greater than the second resolution, and the color image features are used for indicating different brightness regions or different color change regions in the to-be-processed image; inputting the first HDR image into a neural network model for super-resolution processing; performing image enhancement processing on the first HDR image after the super-resolution processing and the color image features through the neural network model, to obtain a second HDR image corresponding to the to-be-processed image, and the second HDR image is an HDR image with the first resolution. The technical scheme of the application can improve the effect of image enhancement processing.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more specifically, to image enhancement methods and apparatus in the field of computer vision. Background Technology

[0002] Computer vision is an integral part of various intelligent / autonomous systems across diverse application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and the military. It's the science of how to use cameras / video cameras and computers to acquire the data and information we need about the objects being photographed. Figuratively speaking, it's about equipping computers with eyes (cameras / video cameras) and a brain (algorithms) to replace human eyes in identifying, tracking, and measuring targets, thus enabling computers to perceive their environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be viewed as the science of how to enable artificial systems to "perceive" from images or multidimensional data. In short, computer vision uses various imaging systems to replace visual organs in acquiring input information, and then the computer replaces the brain in processing and interpreting this input information. The ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, possessing the ability to autonomously adapt to their environment.

[0003] Image enhancement, also known as image quality enhancement, is an important branch of image processing. Image enhancement techniques can improve image quality without re-acquiring data, thus meeting more practical application needs. With the development of deep learning methods, especially those based on convolutional neural networks (CNNs) for image processing, this has become a key driving force in the field of artificial intelligence in recent years, achieving remarkable results in various computer vision tasks, such as image restoration and image enhancement.

[0004] Currently, to meet the real-time requirements of image enhancement for high-resolution images (e.g., 4K resolution), a method combining "downsampling + High Dynamic Range (HDR) + Super-Resolution" has been proposed to save computational overhead and achieve real-time image quality enhancement. This involves downsampling the original resolution input image, performing image enhancement at the lower resolution, and then using a pre-trained CNN to restore the original resolution image based on the original resolution input image and the lower-resolution enhanced image. However, deep learning-based super-resolution methods typically only perform complex super-resolution processes on the luminance channel of the original resolution input image, while simply upsampling the color channels. This leads to inconsistencies in color, brightness, contrast, and saturation between the images before and after super-resolution processing. Therefore, improving the image quality after enhancement while meeting real-time image enhancement requirements is a pressing issue. Summary of the Invention

[0005] This application provides an image enhancement method and an image enhancement apparatus that can enhance the performance of the enhanced image in terms of color, brightness, contrast, saturation, etc., while meeting the real-time requirements of image enhancement processing, thereby improving the effect of image enhancement processing.

[0006] In a first aspect, an image enhancement method is provided, comprising: acquiring a first high dynamic range (HDR) image corresponding to an image to be processed and color image features of the image to be processed, wherein the color image features are used to indicate different brightness regions or different color change regions in the image to be processed, the image to be processed is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; inputting the first HDR image into a neural network model for super-resolution processing; and performing image enhancement processing on the super-resolution processed first HDR image and the color image features through the neural network model to obtain a second HDR image corresponding to the image to be processed, wherein the second HDR image refers to an HDR image with a resolution of the first resolution.

[0007] It should be noted that the second HDR image refers to an image with the same resolution as the image to be processed; where the same resolution means that the image has the same number of pixels. By performing image enhancement on the image to be processed, the pixel values ​​of at least some of the pixels in the image to be processed can be adjusted, so that the visual effect of the second HDR image is better than that of the image to be processed.

[0008] The aforementioned image to be processed can refer to an image that requires image enhancement; for example, the image to be processed can refer to an original image with high resolution but poor image quality; for example, it can refer to an image to be processed that has low image quality due to factors such as weather, distance, and shooting environment; low image quality includes, but is not limited to: image blurring, or poor image color, brightness, saturation, contrast, dynamic range, etc.

[0009] In one possible implementation, the first HDR image can be an image obtained by downsampling the image to be processed and then enhancing it with HDR; wherein, the high dynamic range (HDR) image can provide more dynamic range and image color compared with the standard dynamic range (SDR) image, that is, more image details can be included in the HDR image.

[0010] It should be noted that the aforementioned color image features can also be called color guidance maps. Through color image features, more guidance information can be provided for difficult regions (e.g., bright or dark areas) in the image to be processed in the input neural network model, so that the neural network model can pay more attention to the enhancement effect of difficult regions during the learning process.

[0011] In the embodiments of this application, by acquiring the image to be processed and a first HDR image of the image to be processed, wherein the first HDR image is the low-resolution enhanced image corresponding to the image to be processed, the first HDR image is super-resolution processed in the neural network model using the color image features of the image to be processed, thereby obtaining an enhanced image, namely a second HDR image, with the same resolution size as the image to be processed; by introducing a color attention mechanism, i.e., color image features, when performing super-resolution processing on the first HDR image, the recovery effect of the neural network model on difficult areas (overly bright, overly dark areas) in the image to be processed in terms of color, brightness, contrast, and saturation is improved during the image enhancement process, thereby improving the image enhancement effect.

[0012] In conjunction with the first aspect, some implementations of the first aspect further include: obtaining texture image features of the image to be processed, wherein the texture image features are used to indicate edge regions or texture regions of the image to be processed;

[0013] The step of inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image based on the texture image features using the neural network model.

[0014] It should be noted that the texture attention mechanism is introduced into the neural network model to enable the neural network model to learn details such as edges and textures in the image; by using texture image features, also known as texture guide maps, the neural network model can improve its learning of regions with higher weights in the texture guide maps; thus, the super-resolution process of the first HDR image can improve the super-resolution algorithm's ability to restore image texture details and avoid image blurring or other sensory differences introduced after super-resolution processing.

[0015] In the embodiments of this application, the super-resolution restoration process of the first HDR image can be based on a dual attention mechanism. The dual attention mechanism can include a texture attention mechanism and a color attention mechanism. That is, by using color image features and texture image features in the super-resolution process of the first HDR image, the final enhanced image, i.e., the second HDR image of the image to be processed, is the same as or close to the real image in terms of color, brightness, saturation, contrast and texture details, thereby improving the image enhancement effect.

[0016] In conjunction with the first aspect, some implementations of the first aspect further include: acquiring multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and any one of the multi-scale image features has a different scale.

[0017] The step of inputting the first HDR image into a neural network model for super-resolution processing includes:

[0018] The first HDR image is super-resolution processed using the neural network model based on the multi-scale image features.

[0019] Among them, multi-scale image features can refer to image features of different resolutions. By introducing multi-scale image features, more image detail information can be incorporated into the neural network model, which is beneficial to ensuring the restoration of details in the first HDR image.

[0020] It should be noted that multi-scale image features can be used to indicate the image information of the image to be processed at different scales. Different scales can refer to different resolution sizes, and image information can refer to high-frequency information in the image. For example, high-frequency information can include one or more of the edge information, detail information, and texture information in the image.

[0021] In the embodiments of this application, multi-scale image features of the image to be processed can be obtained, that is, multi-scale feature information of the image to be processed can be extracted from the original resolution of the image to be processed by feature dimensionality reduction; in the super-resolution processing process, multi-scale super-resolution processing restoration can be performed based on the multi-scale features of the first HDR image, i.e., the low-resolution HDR image and the image to be processed, i.e. the original input image. The feature information of the corresponding scale of the low-resolution HDR image and the extracted original input image can be fused to obtain the second HDR image of the image to be processed, i.e., the enhanced image after the image to be processed has been enhanced.

[0022] In conjunction with the first aspect, some implementations of the first aspect further include: obtaining texture image features of the image to be processed, wherein the texture image features are used to indicate edge regions or texture regions of the image to be processed;

[0023] Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different;

[0024] The step of inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image using the neural network model based on the texture image features and the multi-scale features.

[0025] In the embodiments of this application, during the super-resolution processing of the first HDR image, both a dual attention mechanism and multi-scale image features can be introduced. The dual attention mechanism can include a texture attention mechanism and a color attention mechanism, i.e., color image features and texture features. By using the dual attention mechanism and multi-scale image features, the final enhanced image, i.e. the second HDR image of the image to be processed, is the same as or close to the ground image in terms of color, brightness, saturation, contrast, and texture details, thereby improving the image enhancement effect, i.e., improving the image quality after image enhancement processing.

[0026] In conjunction with the first aspect, in certain implementations of the first aspect, the step of performing super-resolution processing on the first HDR image using the neural network model based on the texture image features and the multi-scale image features includes:

[0027] Obtain texture image features at a first scale, wherein the texture image features at the first scale have the same scale as the first scale image features in the multi-scale image;

[0028] The neural network model performs a dot product operation between the first HDR image and the texture image features at the first scale to obtain the third image features.

[0029] The neural network model performs channel combining, convolution, and upsampling operations on the third image features and the first-scale image features.

[0030] In conjunction with the first aspect, in certain implementations of the first aspect, the step of performing image enhancement processing on the first HDR image after super-resolution processing and the color image features through the neural network model to obtain the second HDR image corresponding to the image to be processed includes:

[0031] The second HDR image is obtained by performing dot multiplication and convolution operations on the first HDR image after super-resolution processing and the color image features through the neural network model.

[0032] Secondly, an image enhancement method is provided, applied to a smart screen (or, an AI screen), comprising: detecting a first operation of a user instructing the opening of a display mode interface on the smart screen; responding to the first operation, displaying the display mode selection interface on the smart screen; detecting a second operation of the user instructing a first display mode; and responding to the second operation, displaying an output image on the smart screen, wherein the output image is a second high dynamic range (HDR) image obtained after image enhancement processing of the image to be processed.

[0033] In the image enhancement process, a neural network model is applied to input the first high dynamic range (HDR) image of the image to be processed into the neural network model for super-resolution processing. The second HDR image is obtained by the neural network model through image enhancement processing of the color image features of the first HDR image after super-resolution processing and the image to be processed. The image to be processed is an image with a first resolution, and the first HDR image is an image with a second resolution, wherein the first resolution is greater than the second resolution. The second HDR image refers to an HDR image with a resolution of the first resolution. The color image features are used to indicate different brightness areas or different color change areas in the image to be processed.

[0034] Optionally, the above display mode interface may include, but is not limited to, the following display modes: HDR mode, Dolby mode, Ultra HD mode, Blu-ray mode, and Smart mode.

[0035] It should be understood that the output image described above is a second HDR image obtained after image enhancement processing of the image to be processed, i.e., an HDR image with the same resolution and size as the image to be processed. The image to be processed can refer to an image / video captured by the electronic device through a camera, or it can be an image obtained from within the electronic device (e.g., an image stored in the electronic device's photo album, or a picture obtained from the cloud). For example, the image to be processed can be the original video source stored in a database, or it can refer to a video source being played in real time.

[0036] In one possible implementation, the image enhancement method for the image to be processed described above can be an offline method executed in the cloud. For example, the image enhancement of the image to be processed can be performed by a cloud server to obtain an output image, which can then be displayed on a smart screen.

[0037] In another possible implementation, the image enhancement method described above can be executed by a local device, that is, it can refer to performing image enhancement on the image to be processed on the smart screen to obtain the output image. In one possible implementation, the first operation of the user instructing the smart screen to open the display mode interface may include, but is not limited to: instructing the smart screen to open the display mode interface through a control device, or it may include the user instructing the smart screen to open the display mode interface through voice, or it may include other actions by the user instructing the smart screen to open the display mode interface; the above are illustrative examples and do not limit this application in any way.

[0038] Similarly, the second operation of the user instructing the first display mode may include, but is not limited to: instructing the first display model in the display mode interface through the control device, or the user instructing the first display model in the display mode interface through voice, or the user instructing the first display mode in the display mode interface through other means; the above are for illustrative purposes only and do not limit this application in any way.

[0039] The first display mode mentioned above can refer to HDR mode or other professional mode. The image enhancement method provided in this application embodiment can be implemented through the first display mode, thereby improving the image quality of the output image and enabling users of smart screens to have a better visual experience.

[0040] It should be noted that the specific process for feature enhancement of the image to be processed described above can be obtained from the first aspect and any of its implementation methods.

[0041] In conjunction with the second aspect, some implementations of the second aspect further include: obtaining texture image features of the image to be processed, wherein the texture image features are used to indicate edge regions or texture regions of the image to be processed;

[0042] The step of inputting the first HDR image into the neural network model for super-resolution processing includes: performing super-resolution processing on the first HDR image based on the texture image features using the neural network model.

[0043] In conjunction with the second aspect, some implementations of the second aspect further include: acquiring multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and any one of the multi-scale image features has a different scale.

[0044] The first HDR image is then input into a neural network model for super-resolution processing.

[0045] The first HDR image is super-resolution processed using the neural network model based on the multi-scale image features.

[0046] In conjunction with the second aspect, some implementations of the second aspect further include: obtaining texture image features of the image to be processed, wherein the texture image features are used to indicate edge regions or texture regions of the image to be processed;

[0047] Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different;

[0048] The step of inputting the first HDR image into the neural network model for super-resolution processing includes:

[0049] The neural network model performs super-resolution processing on the first HDR image based on the texture image features and the multi-scale features.

[0050] It should be understood that multi-scale image features can be used to indicate the image information of the image to be processed at different scales. Different scales can refer to different resolution sizes, and image information can refer to high-frequency information in the image. For example, high-frequency information can include one or more of the edge information, detail information, and texture information in the image.

[0051] In conjunction with the second aspect, in certain implementations of the second aspect, the super-resolution processing of the first HDR image based on the texture image features and the multi-scale image features using the neural network model includes:

[0052] Obtain texture image features at a first scale, wherein the texture image features at the first scale have the same scale as the first scale image features in the multi-scale image;

[0053] The neural network model performs a dot product operation between the first HDR image and the texture image features at the first scale to obtain the third image features.

[0054] The neural network model performs channel combining, convolution, and upsampling operations on the third image features and the first-scale image features.

[0055] In conjunction with the second aspect, in certain implementations of the second aspect, the step of performing image enhancement processing on the first HDR image after super-resolution processing and the color image features through the neural network model to obtain the second HDR image corresponding to the image to be processed includes:

[0056] The second HDR image is obtained by performing dot multiplication and convolution operations on the first HDR image after super-resolution processing and the color image features through the neural network model.

[0057] Thirdly, an image enhancement method is provided, applied to an electronic device having a display screen and a camera, comprising: detecting a second operation by a user instructing the camera; in response to the second operation, displaying an output image on the display screen or saving the output image in the electronic device, the output image being a second high dynamic range (HDR) image obtained after image enhancement processing of an image to be processed, wherein a neural network model is applied during the image enhancement processing, and a first HDR image of the image to be processed is input into the neural network model for super-resolution processing; the second HDR image is obtained by the neural network model performing image enhancement processing on the color image features of the first HDR image after super-resolution processing and the image to be processed; the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, the first resolution being greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution; the color image features are used to indicate different brightness areas or different color change areas in the image to be processed.

[0058] Optionally, it may also include: detecting a first operation by the user to open the camera; in response to the first operation, displaying a shooting interface on the display screen, the shooting interface including a viewfinder, the viewfinder including the image to be processed.

[0059] The specific process for performing feature enhancement processing on the image to be processed can be obtained from the first aspect and any of its implementation methods.

[0060] In one possible implementation, the image enhancement method provided in this application embodiment can be applied to the field of photography in smart terminals. The image enhancement method in this application embodiment can improve the performance of the output image obtained by image enhancement processing in terms of color, brightness, contrast, saturation, etc.

[0061] For example, when taking a photo in real time on a smart terminal, the acquired raw image can be enhanced, and the enhanced output image can be displayed on the screen of the smart terminal. Alternatively, the acquired raw image can be enhanced, and the enhanced output image can be saved to the photo album of the smart terminal.

[0062] Fourthly, an image enhancement method is provided, comprising: acquiring a road image to be processed and a first high dynamic range (HDR) image corresponding to the road image, wherein the road image is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; inputting the road image into a neural network model to obtain color image features of the road image, wherein the color image features are used to indicate different brightness areas or different color change areas in the road image; inputting the first HDR image into the neural network model for super-resolution processing; processing the super-resolution processed first HDR image and the color image features through the neural network model to obtain a second HDR image corresponding to the road image, wherein the second HDR image is an HDR image with the first resolution; and identifying road information in the second HDR image based on the second HDR image.

[0063] The specific process for feature enhancement processing of road images described above can be obtained from the first aspect and any of its implementation methods.

[0064] In one possible implementation, the image enhancement method provided in this application can be applied to the field of autonomous driving. For example, it can be applied to the navigation system of autonomous vehicles. Through the image enhancement method in this application, autonomous vehicles can perform image enhancement processing on the original road images with lower image quality during road navigation to obtain enhanced road images, thereby improving the safety of autonomous vehicles.

[0065] Fifthly, an image enhancement method is provided, comprising: acquiring a street scene image to be processed and a first high dynamic range (HDR) image corresponding to the street scene image, wherein the street scene image is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; inputting the street scene image into a neural network model to obtain color image features of the street scene image, wherein the color image features are used to indicate different brightness areas or different color change areas in the street scene image; inputting the first HDR image into the neural network model for super-resolution processing; processing the super-resolution processed first HDR image and the color image features through the neural network model to obtain a second HDR image corresponding to the street scene image, wherein the second HDR image is an HDR image with the first resolution; and identifying street scene information in the second HDR image based on the second HDR image.

[0066] The specific process for feature enhancement of street view images described above can be obtained from the first aspect and any of its implementation methods.

[0067] In one possible implementation, the image enhancement method provided in this application can be applied to the security field. For example, the image enhancement method in this application can be applied to the enhancement of surveillance images in the security field. For instance, images (or videos) captured by surveillance equipment in public places are often affected by factors such as weather and distance, resulting in problems such as blurry images and low image quality. The image enhancement method of this application can enhance the captured original images, thereby recovering important information such as license plate numbers and clear faces, providing crucial clues for case investigation.

[0068] It should be understood that the extensions, limitations, interpretations and descriptions of the relevant content in the first aspect above also apply to the same content in the second, third, fourth and fifth aspects.

[0069] A sixth aspect provides an image enhancement apparatus, including a module / unit for performing the image enhancement method of the first to fifth aspects and any one of the implementations of the first to fifth aspects.

[0070] A seventh aspect provides an image enhancement apparatus, comprising: a memory for storing a program; and a processor for executing the program stored in the memory. When the program stored in the memory is executed, the processor performs the following: acquiring a first high dynamic range (HDR) image corresponding to an image to be processed and color image features of the image to be processed, wherein the color image features are used to indicate different brightness regions or different color change regions in the image to be processed, the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution; inputting the first HDR image into a neural network model for super-resolution processing; and performing image enhancement processing on the super-resolution processed first HDR image and the color image features through the neural network model to obtain a second HDR image corresponding to the image to be processed, wherein the second HDR image refers to an HDR image with a resolution of the first resolution.

[0071] In one possible implementation, the processor included in the image enhancement apparatus is further used for the image enhancement method in the first to fifth aspects and any one of the implementations of the first to fifth aspects.

[0072] Eighthly, a computer-readable medium is provided that stores program code for execution by a device, the program code including the image enhancement method for performing the first to fifth aspects and any one of the implementations of the first to fifth aspects.

[0073] Ninthly, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to execute the image enhancement method in the first to fifth aspects and any one of the implementations of the first to fifth aspects.

[0074] In a tenth aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored in a memory through the data interface to execute the image enhancement method in the first to fifth aspects and any one of the first to fifth aspects.

[0075] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the image enhancement method in the first to fifth aspects and any one of the first to fifth aspects described above. Attached Figure Description

[0076] Figure 1This is a schematic diagram of an artificial intelligence main framework provided in an embodiment of this application;

[0077] Figure 2 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0078] Figure 3 This is a schematic diagram illustrating another application scenario provided by an embodiment of this application;

[0079] Figure 4 This is a schematic diagram illustrating another application scenario provided in the embodiments of this application;

[0080] Figure 5 This is a schematic diagram illustrating another application scenario provided in the embodiments of this application;

[0081] Figure 6 This is a schematic diagram of the system architecture provided in the embodiments of this application;

[0082] Figure 7 This is a schematic diagram of a convolutional neural network structure provided in an embodiment of this application;

[0083] Figure 8 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application;

[0084] Figure 9 This application provides a schematic diagram of a system architecture.

[0085] Figure 10 This is a schematic diagram of the image enhancement processing provided in an embodiment of this application;

[0086] Figure 11 This is a schematic diagram of the system architecture of an image enhancement method provided in an embodiment of this application;

[0087] Figure 12 This is a schematic diagram of the process for HDR correction using a color guide map provided in an embodiment of this application;

[0088] Figure 13 This is a schematic diagram of the system architecture of another image enhancement method provided in the embodiments of this application;

[0089] Figure 14 This is a schematic diagram of the texture guide map extraction process provided in the embodiments of this application;

[0090] Figure 15 This is a schematic diagram of the system architecture of another image enhancement method provided in the embodiments of this application;

[0091] Figure 16 This is a schematic flowchart of the self-guided multi-level super-resolution processing provided in the embodiments of this application;

[0092] Figure 17 This is a schematic diagram of HDR correction using a color guide map after introducing multi-scale image features, as provided in an embodiment of this application.

[0093] Figure 18 This is a schematic diagram illustrating the evaluation results of image enhancement quality provided in the embodiments of this application;

[0094] Figure 19 This is a schematic diagram illustrating the evaluation results of image enhancement quality provided in the embodiments of this application;

[0095] Figure 20 This is a schematic diagram illustrating the evaluation results of image enhancement quality provided in the embodiments of this application;

[0096] Figure 21 This is a schematic block diagram of the image enhancement device provided in the embodiments of this application;

[0097] Figure 22 This is a schematic diagram of the hardware structure of the image enhancement device provided in the embodiments of this application. Detailed Implementation

[0098] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0099] It should be understood that the images in the embodiments of this application can be static images (or static pictures) or moving images (or moving pictures). For example, the images in this application can be videos or animated pictures, or they can be static pictures or photographs. For ease of description, in the following embodiments, static images or moving images will be uniformly referred to as images.

[0100] It should also be understood that in the various embodiments of this application, "first", "second", "third", etc. are only used to refer to different objects and do not indicate any other limitation on the objects referred to.

[0101] Figure 1 A schematic diagram of an artificial intelligence framework is shown, which describes the overall workflow of an artificial intelligence system and is applicable to general artificial intelligence domain needs.

[0102] The above-mentioned artificial intelligence framework 100 will be elaborated in detail from two dimensions: "intelligent information chain" (horizontal axis) and "information technology (IT) value chain" (vertical axis).

[0103] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it could be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom."

[0104] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (provided and processed by technology) to the industrial ecosystem of systems.

[0105] (1) Infrastructure 110

[0106] Infrastructure provides computing power to support artificial intelligence systems, enables them to communicate with the outside world, and provides support through basic platforms.

[0107] Infrastructure can communicate with the outside world through sensors, and its computing power can be provided by smart chips.

[0108] The intelligent chips here can be hardware acceleration chips such as central processing units (CPUs), neural-network processing units (NPUs), graphics processing units (GPUs), application-specific integrated circuits (ASICs), and field-programmable gate arrays (FPGAs).

[0109] The basic platform of the infrastructure can include distributed computing frameworks and related platform guarantees and support, such as cloud storage and computing, and interconnected networks.

[0110] For example, for infrastructure, data can be acquired through sensors and external communication, and then this data can be provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0111] (2) Data 120

[0112] The data at the next layer of infrastructure is used to represent data sources in the field of artificial intelligence. This data includes graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0113] (3) Data processing 130

[0114] The aforementioned data processing typically includes data training, machine learning, deep learning, search, reasoning, and decision-making.

[0115] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0116] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0117] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0118] (4) General ability 140

[0119] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0120] (5) Smart Products and Industry Applications 150

[0121] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent manufacturing, intelligent transportation, smart home, intelligent healthcare, intelligent security, autonomous driving, and intelligent terminals.

[0122] Figure 2 This is a schematic diagram illustrating an application scenario of the image enhancement method provided in the embodiments of this application.

[0123] like Figure 2 As shown, the technical solution of this application embodiment can be applied to smart terminals. The image enhancement method in this application embodiment can perform image enhancement processing on an input image (or input video) to obtain an output image (or output video) after image enhancement.

[0124] The aforementioned smart terminal can be mobile or fixed. For example, the smart terminal can be a mobile phone with image enhancement function, a tablet personal computer (TPC), a media player, a smart TV, a laptop computer (LC), a personal digital assistant (PDA), a personal computer (PC), a camera, a camcorder, a smartwatch, augmented reality (AR) / virtual reality (VR), a wearable device (WD), or an autonomous vehicle, etc. This application embodiment does not limit this.

[0125] It should be noted that in the embodiments of this application, image enhancement can also be referred to as image quality enhancement. Specifically, it can refer to processing the brightness, color, contrast, saturation, and / or dynamic range of an image so that the various indicators of the image meet preset conditions or preset indicators. In the embodiments of this application, image enhancement and image quality enhancement have the same meaning.

[0126] The following examples illustrate specific application scenarios of the embodiments of this application.

[0127] Application Scenario 1: Smart Screen (AI Screen)

[0128] In one embodiment, the image enhancement method of this application can also be applied to the field of smart screens; wherein, smart screens are distinct from traditional televisions and break through the limitations of large-screen categories. With the emergence of new products such as smart screens, mobile phones and smart screens will become the dual centers of users' smart lives in the future. Mobile phones will remain the user's personal center, while smart screens may become the emotional center of the family. Smart screens will assume more roles in the family, not only as the family's audio-visual entertainment center, but also as an information sharing center, control and management center, and multi-device interaction center. For example, when playing videos using smart screens, in order to display better image quality (picture quality), the original video source can be processed using the image enhancement method of this application to improve the picture quality of the source and obtain a better visual experience.

[0129] For example, when playing old movies (movies with older release dates and lower image quality) on a smart screen, the image enhancement method of this application can be used to enhance the image of the old movie, enabling it to display the visual quality of a modern film. For instance, the image enhancement method of this application can be used to enhance the old movie source to a high-dynamic-range (HDR) 10 image or a high-quality video meeting the Dolby Vision standard.

[0130] For example, this application provides an image enhancement method, comprising: acquiring an image to be processed (e.g., the original source of a movie, or an online source) and a first high dynamic range (HDR) image corresponding to the image to be processed, wherein the image to be processed is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; inputting the image to be processed into a neural network model to obtain color image features of the image to be processed, wherein the color image features are used to indicate different brightness areas or different color change areas in the image to be processed; inputting the first HDR image into the neural network model for super-resolution processing; and processing the super-resolution processed first HDR image and the color image features through the neural network model to obtain a second HDR image corresponding to the image to be processed (e.g., a source with improved image quality), wherein the second HDR image is an HDR image with the first resolution.

[0131] For example, this application provides an image enhancement method applied to a smart screen (or, an AI screen), comprising: detecting a first operation of a user instructing the user to open a display mode interface of the smart screen; responding to the first operation, displaying the display mode selection interface on the smart screen; detecting a second operation of the user instructing the user to a first display mode; responding to the second operation, displaying an output image on the smart screen, the output image being a second high dynamic range (HDR) image obtained after image enhancement processing of an image to be processed, wherein a neural network model is applied during the image enhancement processing, inputting the acquired first HDR image of the image to be processed into the neural network model for super-resolution processing; the second HDR image is obtained by the neural network model performing image enhancement processing on the color image features of the first HDR image after super-resolution processing and the image to be processed; the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, the first resolution being greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution; the color image features are used to indicate different brightness areas or different color change areas in the image to be processed.

[0132] Optionally, the above display mode interface may include, but is not limited to, the following display modes: HDR mode, Dolby mode, Ultra HD mode, Blu-ray mode, and Smart mode.

[0133] It should be understood that the output image described above is a second HDR image obtained after image enhancement processing of the image to be processed, i.e., an HDR image with the same resolution and size as the image to be processed. The image to be processed can refer to an image / video captured by the electronic device through a camera, or it can be an image obtained from within the electronic device (e.g., an image stored in the electronic device's photo album, or a picture obtained from the cloud). For example, the image to be processed can be the original video source stored in a database, or it can refer to a video source being played in real time.

[0134] In one possible implementation, the image enhancement method for the image to be processed described above can be an offline method executed in the cloud. For example, the image enhancement of the image to be processed can be performed by a cloud server to obtain an output image, which can then be displayed on a smart screen.

[0135] In another possible implementation, the above image enhancement method can be performed by a local device, that is, it can refer to performing image enhancement on the image to be processed in a smart screen to obtain the output image.

[0136] It should be understood that the first operation of the user instructing the smart screen to open the display mode interface may include, but is not limited to: instructing the smart screen to open the display mode interface through a control device, or it may include the user instructing the smart screen to open the display mode interface through voice, or it may include other actions by the user instructing the smart screen to open the display mode interface; the above are for illustrative purposes only and do not limit this application in any way.

[0137] Similarly, the second operation of the user instructing the first display mode may include, but is not limited to: instructing the first display model in the display mode interface through the control device, or the user instructing the first display model in the display mode interface through voice, or the user instructing the first display mode in the display mode interface through other means; the above are for illustrative purposes only and do not limit this application in any way.

[0138] The first display mode mentioned above can refer to HDR mode or other professional modes. This first display mode enables the image enhancement method provided in this application embodiment to improve the image quality of the output image, thereby providing users of the smart screen with a better visual experience. It should be noted that the image enhancement method provided in this application embodiment is also applicable to the following... Figures 6 to 17The extensions, limitations, explanations, and descriptions of the image enhancement methods in the relevant embodiments are not repeated here.

[0139] Application Scenario 2: Smart Terminal Photography

[0140] In one embodiment, such as Figure 3 As shown, the image enhancement method of this application embodiment can be applied to the shooting of smart terminal devices (e.g., mobile phones). The image enhancement method of this application embodiment can perform image enhancement processing on poor-quality original images (or videos) to obtain an output image (or output video) with improved image quality.

[0141] It should be noted that, in Figure 3 To distinguish it from the grayscale image portion, the color image portion is represented by a diagonal line fill.

[0142] For example, the image enhancement method of this application embodiment can be used to perform image enhancement processing on the original image acquired during real-time photography on a smart terminal, and then display the image enhancement output image on the screen of the smart terminal.

[0143] For example, the image enhancement method of this application embodiment can be used to perform image enhancement processing on the acquired original image, and the image enhancement output image can be saved to the album of the smart terminal.

[0144] For example, this application proposes an image enhancement method applied to an electronic device having a display screen and a camera, comprising: detecting a second operation by a user instructing the camera; in response to the second operation, displaying an output image on the display screen or saving the output image in the electronic device, wherein the output image is a second high dynamic range (HDR) image obtained after image enhancement processing of the image to be processed, wherein a neural network model is applied during the image enhancement processing, and a first HDR image of the image to be processed is input into the neural network model for super-resolution processing; the second HDR image is obtained by the neural network model performing image enhancement processing on the color image features of the first HDR image after super-resolution processing and the image to be processed; the image to be processed is an image of a first resolution, the first HDR image is an image of a second resolution, and the first resolution is greater than the second resolution; the second HDR image refers to an HDR image with a resolution of the first resolution; the color image features are used to indicate different brightness areas or different color change areas in the image to be processed.

[0145] Optionally, it may also include: detecting a first operation by the user to open the camera; in response to the first operation, displaying a shooting interface on the display screen, the shooting interface including a viewfinder, the viewfinder including the image to be processed.

[0146] It should be noted that the image enhancement method provided in the embodiments of this application is also applicable to the following. Figures 6 to 17 The extensions, limitations, explanations, and descriptions of the image enhancement methods in the relevant embodiments are not repeated here.

[0147] Application Scenario 3: Autonomous Driving

[0148] In one embodiment, such as Figure 4 As shown, the image enhancement method of this application embodiment can be applied to the field of autonomous driving. For example, it can be applied to the navigation system of autonomous vehicles. Through the image enhancement method of this application, autonomous vehicles can perform image enhancement processing on the original road image (or original road video) with lower image quality during the navigation process of driving on the road, and obtain the enhanced road image (or road video), thereby realizing the safety of autonomous vehicles.

[0149] For example, this application provides an image enhancement method, including: acquiring a road image to be processed and a first high dynamic range (HDR) image corresponding to the road image, wherein the road image is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution;

[0150] The road image is input into a neural network model to obtain color image features of the road image, wherein the color image features are used to indicate different brightness areas or different color change areas in the road image; the first HDR image is input into the neural network model for super-resolution processing; the first HDR image after super-resolution processing and the color image features are processed by the neural network model to obtain a second HDR image corresponding to the road image, wherein the second HDR image is the first resolution HDR image; based on the second HDR image, road information in the second HDR image is identified.

[0151] It should be noted that the image enhancement method provided in the embodiments of this application is also applicable to the following. Figures 6 to 17 The extensions, limitations, explanations, and descriptions of the image enhancement methods in the relevant embodiments are not repeated here.

[0152] Application Scenario 4: Security Field

[0153] In one embodiment, such as Figure 5As shown, the image enhancement method of this application embodiment can be applied to the security field. For example, the image enhancement method of this application embodiment can be applied to the enhancement of surveillance images in the security field. For instance, images (or videos) captured by surveillance equipment in public places are often affected by factors such as weather and distance, resulting in problems such as blurry images and low image quality. The image enhancement method of this application can enhance the captured images, thereby recovering important information such as license plate numbers and clear faces, providing crucial clues for case investigation.

[0154] For example, this application provides an image enhancement method, comprising: acquiring a street view image to be processed and a first high dynamic range (HDR) image corresponding to the street view image, wherein the street view image is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; inputting the street view image into a neural network model to obtain color image features of the street view image, wherein the color image features are used to indicate different brightness areas or different color change areas in the street view image; inputting the first HDR image into the neural network model for super-resolution processing; processing the super-resolution processed first HDR image and the color image features through the neural network model to obtain a second HDR image corresponding to the street view image, wherein the second HDR image is an HDR image with the first resolution; and identifying street view information in the second HDR image based on the second HDR image.

[0155] It should be noted that the image enhancement method provided in the embodiments of this application is also applicable to the following. Figures 6 to 17 The extensions, limitations, explanations, and descriptions of the image enhancement methods in the relevant embodiments are not repeated here.

[0156] It should be understood that the above are illustrative examples of application scenarios and do not limit the application scenarios of this application in any way.

[0157] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts of neural networks that may be involved in the embodiments of this application will be introduced below.

[0158] (1) Neural Network

[0159] Neural networks can be composed of neural units, which can refer to units represented by x. s The arithmetic unit takes an intercept of 1 as input, and its output can be:

[0160] ;

[0161] Where s = 1, 2, ..., n, n is a natural number greater than 1, W s For x s The weights are denoted by b, where b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce non-linear characteristics into the neural network to convert the input signal into the output signal. The output signal of this activation function can be used as the input to the next convolutional layer; the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple individual neural units, meaning the output of one neural unit can be the input of another. The input of each neural unit can be connected to the local receptive field of the previous layer to extract features from the local receptive field, which can be a region composed of several neural units.

[0162] (2) Deep Neural Networks

[0163] A deep neural network (DNN), also known as a multilayer neural network, can be understood as a neural network with multiple hidden layers. Based on the position of the layers, the internal neural network of a DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer.

[0164] Although DNNs seem complex, the operation of each layer is actually not complicated. Simply put, it involves the following linear relationship expression: ,in, It is the input vector. It is the output vector. It is an offset vector. It is the weight matrix (also called coefficients). It's an activation function. Each layer simply applies the input vector... The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients... and offset vector The number of parameters is also quite large. These parameters are defined in DNNs as follows: [as coefficients] For example: Suppose in a three-layer DNN, the linear coefficient from the fourth neuron in the second layer to the second neuron in the third layer is defined as... The superscript 3 represents the coefficient. The index corresponds to the third-level index 2 in the output and the second-level index 4 in the input.

[0165] In summary, the Lth The coefficients from the k-th neuron in layer 1 to the j-th neuron in layer L are defined as follows: .

[0166] It should be noted that the input layer does not have... In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers in the trained deep neural network (composed of vectors from many layers). The resulting weight matrix.

[0167] (3) Convolutional Neural Network

[0168] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons in a CNN that performs convolutional processing on the input signal. In a convolutional layer of a CNN, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of a series of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are called the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The convolutional kernel can be initialized as a matrix of random size, and during the training process of the CNN, the kernel can learn appropriate weights. Furthermore, the direct benefit of shared weights is that it reduces the connections between layers in the CNN, while also reducing the risk of overfitting.

[0169] (4) Loss function

[0170] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0171] (5) Backpropagation algorithm

[0172] Neural networks can employ backpropagation (BP) to correct the parameters of the initial neural network model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters of the initial neural network model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the neural network model, such as the weight matrix.

[0173] Figure 6 An embodiment of the present application provides a system architecture 200.

[0174] exist Figure 6 In this embodiment, the data acquisition device 260 is used to collect training data. For the image enhancement method of this application, the image enhancement model (also known as an image enhancement network) can be further trained using the training data; that is, the training data collected by the data acquisition device 260 can be training images.

[0175] For example, in the embodiments of this application, the training data for training the image enhancement model may include the original image and the sample enhanced image.

[0176] For example, the original image can refer to an image with lower image quality, while the sample-enhanced image can refer to an image with higher image quality, such as an image that has been improved in one or more aspects such as brightness, color, and detail compared to the sample image.

[0177] It should be understood that image enhancement can also be called image quality enhancement, which specifically refers to processing the brightness, color, contrast, saturation, and / or dynamic range of an image so that the image's various indicators meet preset conditions or preset indicators. In the embodiments of this application, image enhancement and image quality enhancement have the same meaning.

[0178] After collecting the training data, the data acquisition device 260 stores the training data in the database 230. The training device 220 trains the target model / rule 201 (i.e., the image enhancement model in this embodiment) based on the training data maintained in the database 230. The training device 220 inputs the training data into the image enhancement model until the difference between the predicted enhanced image and the sample enhanced image output by the trained image enhancement model meets a preset condition (e.g., the difference between the predicted enhanced image and the sample enhanced image is less than a certain threshold, or the difference between the predicted enhanced image and the sample enhanced image remains unchanged or no longer decreases), thereby completing the training of the target model / rule 201.

[0179] For example, the image enhancement model used to perform the image enhancement method in the embodiments of this application can be trained end-to-end. For instance, the image enhancement model can be trained end-to-end by using the input image and the sample enhancement image (e.g., the ground truth image) corresponding to the input image.

[0180] In the embodiments provided in this application, the target model / rule 201 is obtained by training an image enhancement model. It should be noted that in practical applications, the training data maintained in the database 230 may not all come from the data acquisition device 260; it may also be received from other devices.

[0181] It should also be noted that the training device 220 may not necessarily train the target model / rule 201 entirely based on the training data maintained in the database 230. It may also obtain training data from the cloud or other sources for model training. The above description should not be construed as limiting the embodiments of this application. It should also be noted that at least a portion of the training data maintained in the database 230 may also be used to execute the process of the device 210 processing the data to be processed.

[0182] The target model / rule 201 trained by training device 220 can be applied to different systems or devices, such as... Figure 6The execution device 210 shown can be a terminal, such as a mobile phone terminal, tablet computer, laptop computer, AR / VR, vehicle terminal, etc., or it can be a server or cloud.

[0183] exist Figure 6 In this embodiment, the execution device 210 is configured with an input / output (I / O) interface 212 for data interaction with external devices. Users can input data to the I / O interface 212 through the client device 240. The input data may include the image to be processed input by the client device.

[0184] The preprocessing module 213 and the preprocessing module 214 are used to preprocess the input data (such as the image to be processed) received by the I / O interface 212. In this embodiment, the preprocessing module 213 and the preprocessing module 214 may be omitted (or only one of them may be used), and the calculation module 211 may be used directly to process the input data.

[0185] During the preprocessing of input data by the execution device 210, or during the calculation module 211 of the execution device 210 performing calculations and other related processes, the execution device 210 can call data, code, etc. in the data storage system 250 for corresponding processing, or store the data, instructions, etc. obtained from the corresponding processing into the data storage system 250.

[0186] Finally, the I / O interface 212 returns the processing result, as described above, to the client device 240 to obtain the enhanced image of the image to be processed, i.e., the output image, so as to provide it to the user.

[0187] It is worth noting that the training device 220 can generate corresponding target models / rules 201 based on different training data for different objectives or tasks. The corresponding target models / rules 201 can be used to achieve the above objectives or complete the above tasks, thereby providing the user with the required results.

[0188] exist Figure 6 In the scenario shown, in one case, the user can manually provide input data, which can be done through the interface provided by I / O interface 212.

[0189] In another scenario, client device 240 can automatically send input data to I / O interface 212. If user authorization is required for client device 240 to automatically send input data, the user can set the corresponding permissions in client device 240. The user can view the output results of execution device 210 on client device 240, which can be presented in various forms such as display, sound, or animation. Client device 240 can also act as a data acquisition terminal, collecting the input data and output results of input I / O interface 212 as shown in the figure, and storing them as new sample data in database 230. Alternatively, data can be collected directly from I / O interface 212 without going through client device 240, using the input data and output results of input I / O interface 212 as shown in the figure, and storing them as new sample data in database 230.

[0190] It is worth noting that, Figure 6 This is merely a schematic diagram of a system architecture provided in an embodiment of this application. The positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in Figure 6 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed within the execution device 210.

[0191] like Figure 6 As shown, the target model / rule 201 is obtained by training the training device 220. In this embodiment of the application, the target model / rule 201 can be an image enhancement model. Specifically, the image enhancement model provided in this embodiment of the application can be a deep neural network, a convolutional neural network, or a deep convolutional neural network, etc.

[0192] The following is combined with Figure 7 This section focuses on a detailed introduction to the structure of convolutional neural networks (CNNs). As mentioned in the basic concept introduction above, a CNN is a deep neural network with a convolutional structure. It is a deep learning architecture, which refers to learning at multiple levels of abstraction through machine learning algorithms. As a deep learning architecture, a CNN is a feedforward artificial neural network, in which each neuron can respond to the input image.

[0193] The structure of the image enhancement model in this embodiment can be as follows: Figure 7 As shown. In Figure 7In this example, the convolutional neural network 300 may include an input layer 310, a convolutional / pooling layer 320 (where the pooling layer is optional), a fully connected layer 330, and an output layer 340. The input layer 310 acquires the image to be processed and then passes the acquired image to the convolutional / pooling layer 320 and the fully connected layer 330 for processing to obtain the image processing result. The following section... Figure 7 This section provides a detailed introduction to the internal layer structure of CNN 300.

[0194] 320 convolutional / pooling layers:

[0195] like Figure 7 The convolutional / pooling layer 320 shown may include layers as in Examples 321-326. For example, in one implementation, layer 321 is a convolutional layer, layer 322 is a pooling layer, layer 323 is a convolutional layer, layer 324 is a pooling layer, layer 325 is a convolutional layer, and layer 326 is a pooling layer; in another implementation, layers 321 and 322 are convolutional layers, layer 323 is a pooling layer, layers 324 and 325 are convolutional layers, and layer 326 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0196] The following section will use convolutional layer 321 as an example to introduce the internal working principle of a single convolutional layer.

[0197] Convolutional layer 321 can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During convolution, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, a single weight matrix is ​​not used; instead, multiple weight matrices of the same size (rows × columns) are applied—multiple identical matrices. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image; this dimension can be understood as being determined by the "multiple" mentioned above.

[0198] Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors of the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same size (rows × columns), and the convolutional feature maps extracted by these multiple weight matrices of the same size also have the same size. Then, the multiple convolutional feature maps of the same size are merged to form the output of the convolution operation.

[0199] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, thereby enabling the convolutional neural network 300 to make correct predictions.

[0200] When a convolutional neural network 300 has multiple convolutional layers, the initial convolutional layers (e.g., 321) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 300 increases, the features extracted by later convolutional layers (e.g., 326) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0201] Pooling layer:

[0202] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 7 Layers 321-326 in example 320 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling operators and / or max pooling operators to sample the input image to obtain a smaller image size. The average pooling operator calculates the average value of pixel values ​​within a specific range as the result of average pooling. The max pooling operator selects the pixel with the largest value within a specific range as the result of max pooling.

[0203] Furthermore, just as the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after processing by a pooling layer can be smaller than the size of the input image of the pooling layer. Each pixel in the output image of the pooling layer represents the average or maximum value of the corresponding sub-region of the input image of the pooling layer.

[0204] Fully connected layer 330:

[0205] After processing by the convolutional / pooling layers 320, the convolutional neural network 300 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 320 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 300 needs to utilize fully connected layers 330 to generate one or a set of the required number of classes in the output. Therefore, the fully connected layers 330 can include multiple hidden layers (such as...). Figure 7 As shown in 331, 332 to 33n) and output layer 340, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image enhancement, image recognition, image classification, image detection, and image super-resolution reconstruction, etc.

[0206] After the multiple hidden layers in the fully connected layer 330, the final layer of the entire convolutional neural network 300 is the output layer 340. This output layer 340 has a loss function similar to the classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 300 has propagated forward (e.g., ... Figure 7 Propagation from 310 to 340 degrees is forward propagation, and backward propagation (such as...) is completed. Figure 7 The propagation from 340 to 310 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 300 and the error between the output of the convolutional neural network 300 through the output layer and the ideal result.

[0207] It should be noted that, Figure 7 The convolutional neural network shown is only a structural example of an image enhancement model in this application embodiment. In specific applications, the convolutional neural network used in the image enhancement method of this application embodiment can also exist in the form of other network models.

[0208] In embodiments of this application, the image enhancement device may include Figure 7 The convolutional neural network 300 shown is an image enhancement device that can perform image enhancement processing on the image to be processed to obtain the processed output image.

[0209] Figure 8 This application provides a hardware structure for a chip, which includes a neural network processing unit (NPU) 400. This chip can be configured as follows: Figure 6 The execution device 210 shown is used to perform the calculations of the computing module 211. This chip can also be placed in, for example... Figure 6The training device 220 shown is used to complete the training work of the training device 220 and output the target model / rule 201. For example... Figure 7 The algorithms for each layer in the convolutional neural network shown can all be implemented in, for example... Figure 8 This is achieved in the chip shown.

[0210] The NPU 400 is mounted as a coprocessor on the main central processing unit (CPU), and tasks are assigned by the main CPU. The core of the NPU 400 is the arithmetic circuit 403, and the controller 404 controls the arithmetic circuit 403 to retrieve data from the memory (weight memory or input memory) and perform calculations.

[0211] In some implementations, the arithmetic circuit 403 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 403 is a two-dimensional pulsating array. The arithmetic circuit 403 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 403 is a general-purpose matrix processor.

[0212] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 403 retrieves the corresponding data for matrix B from the weight memory 402 and caches it in each PE (Engineer Component) of the arithmetic circuit 403. The arithmetic circuit 403 retrieves the data for matrix A from the input memory 401 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator 408.

[0213] The vector computation unit 407 can further process the output of the computation circuit 403, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 407 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0214] In some implementations, the vector computation unit 407 can store the processed output vector into a unified memory 406. For example, the vector computation unit 407 can apply a nonlinear function to the output of the arithmetic circuit 403, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 407 generates normalized values, merged values, or both.

[0215] In some implementations, the vector of processed output can be used as an activation input to the computation circuit 403, for example, for use in subsequent layers of the neural network.

[0216] Unified memory 406 is used to store input data and output data. Weight data is directly stored in input memory 401 and / or unified memory 406, weight data in external memory in weight memory 402, and data in unified memory 406 in external memory through direct memory access controller 405 (DMAC).

[0217] The bus interface unit 410 (BIU) is used to enable interaction between the main CPU, DMAC and instruction fetch memory 409 via the bus.

[0218] The instruction fetch buffer 409, connected to the controller 404, is used to store the instructions used by the controller 404. The controller 404 uses the instructions cached in the instruction fetch buffer 409 to control the operation of the arithmetic accelerator.

[0219] Generally, the unified memory 406, input memory 401, weight memory 402, and instruction fetch memory 409 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0220] in, Figure 7 The operations of each layer in the convolutional neural network shown can be performed by the operation circuit 403 or the vector calculation unit 407.

[0221] The above-mentioned Figure 6 The execution device 210 in the embodiment is capable of executing each step of the image enhancement method of this application. Figure 7 The CNN model shown and Figure 8 The chip shown can also be used to perform various steps of the image enhancement method of the embodiments of this application.

[0222] Figure 9The illustration shows a system architecture 500 provided in this application embodiment. The system architecture includes a local device 520, a local device 530, an execution device 510, and a data storage system 550, wherein the local devices 520 and 530 are connected to the execution device 510 through a communication network.

[0223] For example, the execution device 510 may be implemented by one or more servers.

[0224] Optionally, the execution device 510 can be used in conjunction with other computing devices, such as data storage devices, routers, load balancers, etc. The execution device 510 can be deployed at a single physical site or distributed across multiple physical sites. The execution device 510 can use data from the data storage system 550 or call program code from the data storage system 550 to implement the image enhancement method of this application embodiment.

[0225] It should be noted that the aforementioned execution device 510 can also be called a cloud device, in which case the execution device 510 can be deployed in the cloud.

[0226] Specifically, the execution device 510 can perform the following process: acquiring an image to be processed and a first high dynamic range (HDR) image corresponding to the image to be processed, wherein the image to be processed is an image with a first resolution, the first HDR image is an image with a second resolution, and the first resolution is greater than the second resolution; inputting the image to be processed into a neural network model to obtain color image features of the image to be processed, wherein the color image features are used to indicate different brightness areas or different color change areas in the image to be processed; inputting the first HDR image into the neural network model for super-resolution processing; and processing the first HDR image after super-resolution processing and the color image features through the neural network model to obtain a second HDR image corresponding to the image to be processed, wherein the second HDR image is an HDR image with the first resolution.

[0227] In one possible implementation, the image enhancement method of this application embodiment can be an offline method executed in the cloud. For example, the image enhancement method of this application embodiment can be executed in the execution device 510 described above.

[0228] In one possible implementation, the image enhancement method of this application embodiment may be executed by local device 520 or local device 530.

[0229] In the embodiments of this application, image enhancement can be performed on the acquired image to be processed with poor image quality, thereby obtaining an output image, namely a second HDR image, in which the performance of the image to be processed is improved in terms of image detail, image color and image brightness.

[0230] For example, users can interact with execution device 510 by operating their respective user devices (e.g., local device 520 and local device 530). Each local device can represent any computing device, such as a personal computer, computer workstation, smartphone, tablet, smart camera, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0231] Each user's local device can interact with the execution device 510 through a communication network of any communication mechanism / standard. The communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof.

[0232] In one implementation, local devices 520 and 530 can obtain the relevant parameters of the aforementioned neural network model from execution device 510, deploy the neural network model on local devices 520 and 530, and use the neural network model for image enhancement processing, etc.

[0233] In another implementation, a neural network model can be directly deployed on the execution device 510. The execution device 510 obtains the image to be processed and the first HDR image corresponding to the image to be processed from the local device 520 and the local device 530, and performs image enhancement processing according to the neural network model to obtain the second HDR image corresponding to the image to be processed.

[0234] For example, the neural network model described above could be a super-resolution network as described in the embodiments of this application, such as, see below. Figure 11 , Figure 13 as well as Figure 15 The super-resolution network 720 in the middle.

[0235] Currently, to achieve real-time enhancement of high-resolution images, a "downsampling + HDR + super-resolution" approach has been proposed to save computational overhead and achieve real-time image quality enhancement. The original resolution image and the low-resolution enhanced image can be input into a pre-trained convolutional neural network to perform super-resolution processing (i.e., restoring the original resolution of the image), resulting in an output original-resolution enhanced image. However, this approach to high-resolution image enhancement has two drawbacks. First, to save computational overhead, deep learning-based super-resolution methods typically only perform complex super-resolution processes on the brightness channels of the image, while simple upsampling can be used for the color channels. This leads to a failure to guarantee consistency in color, brightness, contrast, saturation, and dynamic range between the original-resolution enhanced image and the low-resolution enhanced image. Second, the super-resolution process based on the original resolution image and the low-resolution enhanced image is not explained in detail in the above image enhancement methods. Simple super-resolution processing can lead to problems such as image blurring and loss of texture details; while overly complex super-resolution processes can increase the processing time of the entire convolutional neural network and may introduce problems such as checkerboard patterns and artifacts.

[0236] In view of this, embodiments of this application provide an image enhancement method and an image enhancement apparatus. The method involves acquiring an image to be processed and a first HDR image of the image to be processed, wherein the first HDR image is a low-resolution enhanced image corresponding to the image to be processed. In a neural network model, the first HDR image is super-resolution processed using the color image features of the image to be processed, thereby obtaining an enhanced image, i.e., a second HDR image, with the same resolution as the image to be processed. In this embodiment, a color attention mechanism is introduced on the basis of "downsampling + HDR + super-resolution," that is, color image features are used to improve the recovery effect of the neural network model on difficult areas (overly bright or overly dark areas) in the image to be processed in terms of color, brightness, contrast, and saturation during the image enhancement process, thereby improving the image enhancement effect.

[0237] Figure 10 A schematic flowchart of the image enhancement method provided in an embodiment of this application is shown. Figure 10 The image enhancement method shown can be performed by an image enhancement device, which can specifically be... Figure 6 The execution device 210 in the middle can also be Figure 9 The execution device 510 or the local device. Figure 10 The method shown includes steps 610 to 630, which will be described in detail below.

[0238] Step 610: Obtain the first high dynamic range (HDR) image corresponding to the image to be processed and the color image features of the image to be processed.

[0239] The color image features are used to indicate different brightness areas or different color change areas in the image to be processed. The image to be processed is an image with a first resolution, and the first HDR image is an image with a second resolution. The first resolution is greater than the second resolution.

[0240] For example, the image to be processed mentioned above can refer to an image that requires image enhancement; for example, the image to be processed can refer to an original image with high resolution but poor image quality; for example, it can refer to an image to be processed that has low image quality due to factors such as weather, distance, and shooting environment; low image quality includes, but is not limited to: low image quality, including but not limited to image blurring, or poor image color, brightness, saturation, contrast, dynamic range, etc.

[0241] For example, the image to be processed could be an image captured by the electronic device's camera, or it could be an image obtained from within the electronic device itself (e.g., an image stored in the electronic device's photo album, or an image retrieved from the cloud by the electronic device). For example, the electronic device could be... Figure 9 Either the local device or the execution device shown.

[0242] For example, the first HDR image may be an image obtained by downsampling the image to be processed and then enhancing it with HDR; wherein, a high dynamic range (HDR) image can provide more dynamic range and image color than a standard dynamic range (SDR) image, that is, more image details can be included in an HDR image.

[0243] In the embodiments of this application, the color image features of the image to be processed can be obtained through target learning or traditional methods.

[0244] For example, the color image features of the image to be processed can be obtained by using target learning or traditional methods.

[0245] The core of the target learning method lies in providing a learning target that allows us to measure the difference between the input image and the ground truth image (the learning target). This difference is usually reflected in the bright and shadow (dark) areas of the image.

[0246] In addition, traditional methods can be used to obtain the color image features of the image to be processed. In this case, the ground truth image is no longer needed, and there is no learning process. For example, the color input image can be converted to a grayscale image or converted to other color domains with a brightness channel (such as the YUV domain or the LAB domain). Then, a minimum threshold and a maximum threshold are set. The parts of the channel (such as grayscale, the Y channel in the YUV domain, and the L channel in the LAB domain) that are below and above the minimum and maximum thresholds are designated as regions with higher weights, while other regions are designated as regions with lower weights.

[0247] In one example, the color image features of the image to be processed can be obtained by passing the image to be processed through an autoencoder / decoder subnetwork.

[0248] In one example, due to limitations in terminal device speed, video memory, and power consumption, to save computation, the image to be processed can be downsampled to obtain a low-resolution image. Color image features are then obtained by combining the low-resolution image with the autoencoder / decoder subnetwork. See below for an example. Figure 12 The color image features of the image to be processed are obtained from a low-resolution image of the image to be processed.

[0249] It should be noted that the aforementioned color image features can also be called color guidance maps. Through color image features, more guidance information can be provided for difficult regions (e.g., bright or dark areas) in the image to be processed in the input neural network model, so that the neural network model can pay more attention to the enhancement effect of difficult regions during the learning process.

[0250] Step 620: Input the first HDR image into the neural network model for super-resolution processing.

[0251] The aforementioned super-resolution processing can refer to convolution and upsampling operations, thereby making the resolution of the first HDR image the same as that of the image to be processed.

[0252] Furthermore, in order to make the neural network model pay more attention to edge regions and texture features when performing super-resolution processing on the first HDR image, a texture attention mechanism can be introduced when performing super-resolution processing on the first HDR image.

[0253] Optionally, in one possible implementation, the image to be processed can be input into the neural network model to obtain the texture image features of the image to be processed, wherein the texture image features can be used to indicate the edge regions and texture regions of the image to be processed; inputting the first HDR image into the neural network model for super-resolution processing can refer to performing super-resolution processing on the first HDR image based on the texture image features through the neural network model.

[0254] It should be noted that the texture attention mechanism is introduced into the neural network model to enable the neural network model to learn details such as edges and textures in the image; by using texture image features, also known as texture guide maps, the neural network model can improve its learning of regions with higher weights in the texture guide maps; thus, the super-resolution process of the first HDR image can improve the super-resolution algorithm's ability to restore image texture details and avoid image blurring or other sensory differences introduced after super-resolution processing.

[0255] Furthermore, during the super-resolution processing of the first HDR image, multi-scale image features of the image to be processed can be introduced. These multi-scale image features can refer to image features of different resolutions. By introducing multi-scale image features, more image detail information can be incorporated into the neural network model, which is beneficial to ensuring the restoration of details in the first HDR image.

[0256] In the embodiments of this application, the super-resolution restoration process for the first HDR image can be based on a dual attention mechanism, which may include a texture attention mechanism and a color attention mechanism. By using the color attention mechanism and texture attention mechanism in the super-resolution process, the resulting enhanced image, i.e., the second HDR image of the image to be processed, is the same as or close to the ground image in terms of color, brightness, saturation, contrast, and texture details, thereby improving the image enhancement effect.

[0257] Optionally, in one possible implementation, the image to be processed can be input into a neural network model to obtain multi-scale image features of the image to be processed, wherein any one of the multi-scale image features has a different scale. The neural network model can then perform super-resolution processing on the first HDR image based on the texture image features and the multi-scale image features, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales. For example, see the following... Figure 16 The diagram shown is a schematic flowchart of a self-guided multi-stage super-resolution process.

[0258] It should be understood that multi-scale image features can be used to indicate the image information of the image to be processed at different scales. Different scales can refer to different resolution sizes, and image information can refer to high-frequency information in the image. For example, high-frequency information can include one or more of the edge information, detail information, and texture information in the image.

[0259] For example, super-resolution processing of the first HDR image using a neural network model based on texture image features and multi-scale image features may include: performing scale adjustment processing on the texture image features using a neural network model to obtain texture image features at a first scale, wherein the texture image features at the first scale have the same scale as the first scale image features in the multi-scale image; performing a dot product operation on the first HDR image and the texture image features at the first scale using a neural network model to obtain third image features; and performing channel combining, convolution, and upsampling operations on the third image features and the first scale image features using a neural network model.

[0260] In the embodiments of this application, a multi-level super-resolution processing method is proposed, which extracts multi-scale feature information from the original resolution input image by feature dimensionality reduction. In the super-resolution processing process, multi-scale super-resolution processing restoration can be performed based on the multi-scale features of the first HDR image (i.e., the low-resolution HDR image) and the image to be processed (i.e., the original input image). The feature information of the corresponding scale of the low-resolution HDR image and the extracted original input image can be fused to obtain the second HDR image of the image to be processed, i.e., the enhanced image after the image to be processed has been enhanced.

[0261] Step 630: Process the features of the first HDR image and the color image after super-resolution processing using a neural network model to obtain the second HDR image corresponding to the image to be processed.

[0262] The second HDR image is an HDR image with the first resolution, meaning that the second HDR image has the same resolution as the image to be processed. The second HDR image can refer to the enhanced image obtained after enhancing the image to be processed.

[0263] For example, a second HDR image can be obtained by performing dot multiplication and convolution operations on the features of the first HDR image after super-resolution processing and the color image through a neural network model.

[0264] The dot product operation mentioned above can refer to multiplication on a pixel-by-pixel basis.

[0265] Figure 11 This is a schematic diagram of the system architecture of an image enhancement method provided in an embodiment of this application. For example... Figure 11 As shown, the system architecture may include an HDR network 710 and a super resolution (SR) network 720; wherein, the HDR network 710 may include a downsampling unit 711 and an HDR enhancement unit 712; the SR network 720 may include a color attention unit 721, a super-resolution processing unit 722 and an HDR correction unit 723.

[0266] For example, the downsampling unit 711 described above can be used to downsample an input image, such as a high-resolution image, to obtain a low-resolution image or a small-resolution image, such as a 1080P resolution image.

[0267] For example, the HDR enhancement unit 712 described above can be used to process the downsampled low-resolution image using an HDR enhancement method to obtain a low-resolution HDR enhanced image.

[0268] HDR enhancement methods can employ any neural network model with image enhancement capabilities for image enhancement processing; for example, various neural networks based on UNet or High Dynamic Range Network (HDRNet) frameworks can be used.

[0269] For example, the color attention unit 721 described above can be used to extract a color guide map of an input image (e.g., an original resolution image) using a certain calculation method; wherein, the color guide map can be a color image feature corresponding to the input image.

[0270] In the embodiments of this application, the color image features of the image to be processed can be obtained through target learning or traditional methods.

[0271] For example, the color image features of the image to be processed can be obtained by using target learning or traditional methods.

[0272] The core of the target learning method lies in providing a learning target that allows us to measure the difference between the input image and the ground truth image (the learning target). This difference is usually reflected in the bright and shadow (dark) areas of the image.

[0273] In addition, traditional methods can be used to obtain the color image features of the image to be processed. In this case, the ground truth image is no longer needed, and there is no learning process. For example, the color input image can be converted to a grayscale image or converted to other color domains with a brightness channel (such as the YUV domain or the LAB domain). Then, a minimum threshold and a maximum threshold are set. The parts of the channel (such as grayscale, the Y channel in the YUV domain, and the L channel in the LAB domain) that are below and above the minimum and maximum thresholds are designated as regions with higher weights, while other regions are designated as regions with lower weights.

[0274] It should be understood that the aforementioned color guidance map has higher guidance information for difficult regions (e.g., bright or dark areas) in the input image, allowing the SR Network 720 to focus more on enhancing difficult regions during the learning process.

[0275] For example, the HDR correction unit 723 can be used to correct the image after super-resolution processing by combining the low-resolution enhanced image output by the HDR enhancement unit 712 and the image output by the super-resolution processing unit 722 with the color guide map obtained by the color attention unit 721, and use CNN to correct the color, brightness, contrast and saturation of the super-resolution image, thereby ensuring the consistency of the HDR effect before and after the super-resolution processing unit 722 and strengthening the enhancement effect on difficult areas of the image.

[0276] The following is combined with Figure 12 The process of HDR correction using color guide maps is described in detail. Figure 12 This is a schematic diagram of HDR correction using a color guide map, provided in an embodiment of this application.

[0277] in, Figure 12 The color guide map shown, i.e., the color image features corresponding to the input image, can be extracted using a pre-trained network. For example, an Encoder-Decoder Network (Encoder-Decoder Network) can be used as the network for extracting the color guide map. The input image of the Encoder-Decoder Network can be a low-resolution input image. The color guide map can be obtained after passing through the Encoder-Decoder Network. In the pre-training process, the Encoder-Decoder Network can design an objective function to ensure that difficult regions (e.g., bright or dark areas) in the input image have greater weights in the color guide map.

[0278] For example, the following objective function can be used to train an autoencoder / decoder network:

[0279] ;

[0280] in, This indicates finding the maximum value across the channel dimensions of the input image; A map Represents a color guide image; I in This represents the input image features of the autoencoder / decoder network; for example, it could refer to features of the original resolution image or features of a low-resolution image. target This represents the ground truth image with the same resolution as the input image.

[0281] The objective function shown above allows the autoencoder / decoder network to assign greater weight to regions in the color guide map where the difference between the input and target images is greater.

[0282] During the training of the SR network 720, the color guide map extraction network, for example... Figure 12The autoencoder / decoder network shown can no longer participate in training; in darker areas of the input image and areas with more color changes, the corresponding color guide map can show greater weight, thereby guiding the backend HDR correction unit 723 to focus on learning these areas.

[0283] In one example, such as Figure 12 As shown, the input image can be the original resolution image; or, to save computation, the input image can be a low-resolution image obtained by downsampling the original resolution image.

[0284] It should be noted that if the color guide map is obtained from the original resolution image, then the color guide map may not need to undergo upsampling.

[0285] For example, the HDR correction unit 723 can be a pre-trained CNN network. The HDR correction unit 723 can receive three parts of input data. The first input data is a first original resolution enhanced image output by the super-resolution processing unit 722. The first original resolution enhanced image can refer to the image obtained after upsampling the low-resolution enhanced image output by the HDR enhancement unit 712. The second input data is the low-resolution enhanced image output by the HDR enhancement unit 712. The third input data is the color attention guide map output by the color attention unit 721. The HDR correction unit 723 performs HDR correction based on the above three parts of input data and outputs a second original resolution enhanced image.

[0286] Furthermore, to enable the super-resolution processing unit 722 to focus more on learning the texture and edges in the input image during super-resolution processing, a texture attention unit can be introduced into the system architecture, such as... Figure 13 As shown.

[0287] Figure 13 The super-resolution processing unit 722 may also include a texture attention unit 724, which can be used to extract a texture guide map (i.e., high-frequency information of the image) based on the input image, i.e., the original resolution image, using a certain calculation method. When performing upsampling and convolution operations, the super-resolution processing unit 722 can also use the texture guide map output by the texture attention unit 724 to enhance the restoration of image detail texture and edge regions during the super-resolution process.

[0288] The following is combined with Figure 14 The process of obtaining the texture guide map is described. Figure 14 This is a schematic diagram of the texture guide map extraction process provided in the embodiments of this application.

[0289] For example, the input data of the texture attention unit can be an input image without color information, such as a grayscale image or the Y channel of the YUV color space; for instance, the input data is Y channel image features, which can be filtered out by Gaussian filtering to remove high-frequency information in the image. The output data obtained after filtering is negatively residuald with the input image to obtain the high-frequency information of the image, i.e., the texture guide map. The texture and edges in the image are usually present in the high-frequency information of the image, such as... Figure 14 As shown.

[0290] It should be noted that the process of obtaining the texture guide map is not limited to this method. Edge detection and other algorithms can also be used to obtain the texture guide map. This application does not impose any restrictions on this.

[0291] In the embodiments of this application, the texture attention unit 724 can be used to extract high-frequency information in the image as a texture guide map. The extracted texture guide map will be applied to the super-resolution processing unit 722, thereby improving the learning of the SR network 200 for the regions with higher weights in the guide map and enhancing the learning of the SR network 720 for details such as image edges and textures.

[0292] In one example, the super-resolution processing unit described above can employ a self-guided multi-stage super-resolution unit, meaning the super-resolution process can involve multiple image sizes progressively increasing to the same size as the original resolution input image, such as... Figure 15 As shown.

[0293] Figure 15 This is a schematic diagram of the system architecture of the image enhancement method provided in the embodiments of this application. For example... Figure 15 As shown, the system architecture may include an HDR network 710 and a super resolution (SR) network 720; wherein, the HDR network 710 may include a downsampling unit 711 and an HDR enhancement unit 712; the SR network 720 may include a color attention unit 721, a self-guided multi-level super-resolution unit 722, an HDR correction unit 723, a texture attention unit 724, and a multi-scale self-guided feature extraction unit 725.

[0294] The aforementioned HDR network 710 is used to perform HDR enhancement on the input image I (e.g., 4K resolution) based on the original resolution of the input image, thereby obtaining a low-resolution HDR image (e.g., 1080P resolution) after HDR enhancement.

[0295] For example, the input image can be an original resolution image, such as a high-resolution image or a full-resolution image; the input image is processed by the downsampling unit 711 to obtain a low-resolution image or a small-resolution image; the low-resolution image is input into the HDR enhancement unit 712 for processing to obtain an output low-resolution HDR image.

[0296] The SR network 720 is used to perform super-resolution processing on the input low-resolution HDR image and the original resolution image through the color attention unit 721 and the multi-scale self-guided feature extraction unit 725 included in the SR network 720, thereby restoring the original resolution of the image and obtaining the output original resolution enhanced image (e.g., 4K resolution).

[0297] For example, such as Figure 15 As shown, the multi-scale self-guided feature extraction unit 725 in the SR network 720 can be used to extract features from the input original resolution image through a convolutional neural network, thereby obtaining self-guided maps at multiple scales.

[0298] It should be noted that the aforementioned self-guided graphs at multiple scales can refer to image features at different scales obtained by downsampling and convolution at different depths from the input image. The downsampling method can include, but is not limited to, sampling interpolation methods or pixel rearrangement (space to depth) operations, and multiple scales can refer to multiple resolution sizes.

[0299] For example, the color attention unit 721 described above can be used to extract a color guide map of an input image (e.g., an original resolution image) using a certain calculation method; wherein, the color guide map can be a color image feature corresponding to the input image.

[0300] It should be understood that the aforementioned color guidance map has higher guidance information for difficult regions (e.g., bright or dark areas) in the input image, allowing the SR Network 720 to focus more on enhancing difficult regions during the learning process.

[0301] For example, the texture attention unit 724 described above can be used to extract a texture guide map (i.e., high-frequency information of the image) based on the input image, i.e., the original resolution image, using a certain calculation method. The specific process can be found in the above description. Figure 14 This will not be elaborated upon here.

[0302] For example, such as Figure 15 As shown, the input data in the self-guided multi-level super-resolution unit 722 may include image features of different scales input by the multi-scale self-guided feature extraction unit 725, texture image features corresponding to the input image output by the texture attention unit 724, and the low-resolution HDR enhanced image input by the HDR enhancement unit 712. The self-guided multi-level super-resolution unit 722 can perform upsampling and convolution operations on the above input data to restore the image resolution, thereby obtaining the super-resolution restored image H. SR .

[0303] It should be noted that the introduction of texture attention image features in the above super-resolution process can enhance the recovery of image details, textures and edge regions during super-resolution; in addition, inputting multi-scale image features can make the input image details and information more abundant during the super-resolution process.

[0304] For example, the HDR correction unit 723 described above can be used to correct the super-resolution restored image HDR. SR Corrections are made to color, brightness, contrast, and saturation to obtain an enhanced image of the original resolution, making it infinitely close to the true image. The input data to the HDR correction unit 723 can be the super-resolution restored image H... SR Low-resolution HDR images L In addition to color attention image features, the input data is convolved to obtain image features that are the same as or deviate from the ground image features within a preset range.

[0305] For example, Figure 16 This is a schematic flowchart of the self-guided multi-level super-resolution processing provided in the embodiments of this application.

[0306] like Figure 16 As shown, assuming the input original resolution image is W×H, the multi-scale guided image obtained through downsampling and feature extraction is an image feature with a resolution of W / 2×H / 2 and an image feature with a resolution of W / 4×H / 4. The multi-scale guided image can be used as prior information and concatenated with the output features of each stage of the super-resolution process to guide the super-resolution process. For example, for a resolution of W / 2×H / 2, the texture attention image feature is first scaled to obtain a texture attention image feature with a resolution of W / 4×H / 4. The texture attention image feature of W / 4×H / 4 is then multiplied with the HDR enhanced image with a resolution of W / 4×H / 4, which can be understood as pixel-wise multiplication. Finally, the image feature obtained after the multiplication is multiplied with the original resolution input image. The image features with a resolution of W / 4×H / 4 are concatenated, i.e., channel merging. Similarly, for a resolution of W / 2×H / 2, the texture attention image features are first scaled to obtain texture attention image features with a resolution of W / 2×H / 2. The texture attention image features with a resolution of W / 2×H / 2 are then multiplied by a dot product, i.e., multiplied pixel by pixel. The image features obtained after the dot product are then concatenated with the image features of the original resolution input image with a resolution of W / 2×H / 2, i.e., channel merging. Finally, the image features with a resolution of W / 2×H / 2 after channel merging are upsampled and convolved to obtain the original resolution enhanced image.

[0307] It should be understood that the above is an example of a three-stage progressive super-resolution process with resolutions of W / 4×H / 4, W / 2×H / 2, and W×H. Other multi-stage progressive restorations are also possible, and this application does not limit them in any way.

[0308] For example, Figure 17 The diagram illustrates HDR correction using a color guide map after introducing multi-scale image features. Figure 17 As shown, super-resolution of a YUV domain image is described in detail as an example. The low-resolution image can refer to the U and V channel image features of a low-resolution HDR image, while the original resolution image can refer to the Y channel image features extracted from the image output by the self-guided multi-level super-resolution unit 722. The U and V channel image features of the low-resolution image are upsampled to make their scale the same as the original resolution image. Then, the upsampled U and V channel image features are concatenated with the Y channel image features of the original resolution image (channel merging). The merged YUV image is then converted to an RGB image. The color guide image is upsampled to make its scale the same as the original resolution image. The converted RGB image and the upsampled color guide image are multiplied pixel-by-pixel. Finally, HDR correction (convolution) is performed on the multiplied image to ensure that the difference between the final output original resolution enhanced image and the ground truth image is less than a preset threshold.

[0309] It should be noted that if the color guide map is obtained from the original resolution image, then the color guide map may not need to undergo upsampling.

[0310] It should be understood that Figure 17 Self-encoder / decoder network and Figure 12 The same can be used in the above, which will not be elaborated here; similarly, the HDR correction unit can also be used with... Figure 12 The HDR correction unit is the same as that in the previous one, see [link / reference]. Figure 12 The description in the text will not be repeated here.

[0311] Table 1

[0312]

[0313] Table 1 shows the performance test results of the image enhancement method and benchmark algorithm in the embodiments of this application. Table 1 shows the performance test results of the model in this application and several existing models when performing image enhancement processing. The test metrics are Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), and computational complexity. The computational complexity can be used to represent the number of multiply-accumulate (MAC) operations, where 1G = 10^ 9 .

[0314] As shown in Table 1, the test results demonstrate that the neural network model proposed in this scheme significantly outperforms the range scaling global U-Net (RSGUnet) and the self-guided network (SGN) in terms of quantitative metrics PSNR and SSIM. Furthermore, compared to HDRNet, with PSNR and SSIM losses of 1.85% and 0.26% respectively, the processing time per frame is reduced by 56.2%, and the computational cost per frame is reduced by 86.6%. In the same test environment, it can be seen that the image enhancement method provided in this application has superior processing speed and computational overhead, simultaneously meeting the requirements for image quality enhancement and real-time processing.

[0315] Figures 18 to 20 This is a schematic diagram illustrating the evaluation results of visual quality image enhancement quality provided in an embodiment of this application. Wherein, Figure 18 (a) in the image is the input image, i.e., the image to be processed. Figure 18 (b) in the image is the ground truth image corresponding to the input image; Figure 18 (c) in the image is the output image obtained by enhancing the input image using the image enhancement method of this application; Figure 18 In the image, (d) represents the output image obtained by enhancing the input image using the HDRNet model; Figure 18 (e) in the image represents the output image obtained by enhancing the input image using the RSGUnet model. Figure 18 In the image, (f) represents the output image obtained by enhancing the input image using the SGN model; from Figure 18 (a) to Figure 18 As can be seen from (f) in the example, the output image obtained through the embodiments of this application is closest to the true image, and there are no artifacts or other problems; similarly, through Figure 19 (a) to Figure 19As can be seen from (f) in the figure, the output image obtained through the embodiments of this application can reduce the halo problem in the sky, that is, the image enhancement method of the embodiments of this application can reduce the halo problem while ensuring the HDR effect; similarly, through Figure 20 (a) to Figure 20 As can be seen from (f) in the example, the output image obtained through the embodiment of this application has a color that is closer to the true image for the highlighted parts, and the texture details of the leaf veins are restored more clearly.

[0316] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0317] The above text combined Figures 1 to 20 The image enhancement method provided in the embodiments of this application is described in detail below. Figure 21 and Figure 22 This document describes in detail the apparatus embodiments of this application. It should be understood that the image enhancement apparatus in the embodiments of this application can perform the various image enhancement methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0318] Figure 21 This is a schematic block diagram of an image enhancement apparatus provided in an embodiment of this application. It should be understood that the image enhancement apparatus 800 can perform... Figure 10 The image enhancement method shown is illustrated. The image enhancement apparatus 800 includes an acquisition unit 810 and a processing unit 820.

[0319] The acquisition unit 810 is used to acquire a first high dynamic range (HDR) image corresponding to the image to be processed and the color image features of the image to be processed. The image to be processed is an image with a first resolution, and the first HDR image is an image with a second resolution. The first resolution is greater than the second resolution. The color image features are used to indicate different brightness areas or different color change areas in the image to be processed. The processing unit 820 is used to input the first HDR image into a neural network model for super-resolution processing. The neural network model performs image enhancement processing on the first HDR image after super-resolution processing and the color image features to obtain a second HDR image corresponding to the image to be processed. The second HDR image refers to an HDR image with a resolution of the first resolution.

[0320] Optionally, as an embodiment, the acquisition unit 810 is further configured to:

[0321] Obtain the texture image features of the image to be processed, wherein the texture image features are used to indicate the edge region or texture region of the image to be processed;

[0322] The processing unit 820 is specifically used for:

[0323] The neural network model performs super-resolution processing on the first HDR image based on the texture image features.

[0324] Optionally, as an embodiment, the acquisition unit 810 is further configured to:

[0325] Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different;

[0326] The processing unit 820 is specifically used for:

[0327] The first HDR image is super-resolution processed using the neural network model based on the multi-scale image features.

[0328] Optionally, as an embodiment, the acquisition unit 810 is further configured to:

[0329] The texture image features of the image to be processed are obtained, wherein the texture image features are used to indicate the edge region or texture region of the image to be processed; the multi-scale image features of the image to be processed are obtained, wherein the multi-scale image features are used to indicate the image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different.

[0330] The processing unit 820 is specifically used for:

[0331] The neural network model performs super-resolution processing on the first HDR image based on the texture image features and the multi-scale features.

[0332] Optionally, as an embodiment, the acquisition unit 810 is specifically used for:

[0333] Obtain texture image features at a first scale, wherein the texture image features at the first scale have the same scale as the first scale image features in the multi-scale image;

[0334] The processing unit 820 is specifically used for:

[0335] The neural network model performs a dot product operation on the first HDR image and the texture image features at the first scale to obtain the third image features; the neural network model then performs channel combining, convolution, and upsampling operations on the third image features and the image features at the first scale.

[0336] Optionally, as an embodiment, the processing unit 820 is specifically used for:

[0337] The second HDR image is obtained by performing dot multiplication and convolution operations on the first HDR image after super-resolution processing and the color image features through the neural network model.

[0338] It should be noted that the image enhancement device 800 described above is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, and there is no specific limitation on this.

[0339] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0340] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0341] Figure 22 This is a schematic diagram of the hardware structure of the image enhancement device provided in an embodiment of this application. For example... Figure 22 The image enhancement device 900 shown (which may specifically be a computer device) includes a memory 901, a processor 902, a communication interface 903, and a bus 904. The memory 901, processor 902, and communication interface 903 are interconnected via the bus 904.

[0342] The memory 901 can be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 901 can store a program. When the program stored in the memory 901 is executed by the processor 902, the processor 902 performs various steps of the image enhancement method of this embodiment, for example, executing... Figures 10 to 17 The steps shown.

[0343] It should be understood that the image enhancement device shown in the embodiments of this application can be a server, for example, a server in the cloud, or a chip configured in a server in the cloud; or, the image enhancement device shown in the embodiments of this application can be a smart terminal, or a chip configured in a smart terminal.

[0344] The image enhancement method disclosed in the above embodiments of this application can be applied to, or implemented by, processor 902. Processor 902 may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above image enhancement method can be completed by integrated logic circuits in the hardware of processor 902 or by instructions in software form. For example, processor 902 may contain... Figure 8 The NPU chip shown.

[0345] The processor 902 mentioned above can be a central processing unit (CPU), graphics processing unit (GPU), general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory 901. The processor 902 reads the instructions in memory 901 and, in conjunction with its hardware, completes the implementation of this application. Figure 21 The image enhancement apparatus shown includes units that are required to perform functions, or to perform the methods described in this application. Figures 10 to 17 The steps of the image enhancement method shown are as follows.

[0346] The communication interface 903 uses transceiver devices, such as, but not limited to, transceivers, to enable communication between the device 900 and other devices or communication networks.

[0347] Bus 904 may include a pathway for transmitting information between various components of the image enhancement device 900 (e.g., memory 901, processor 902, communication interface 903).

[0348] It should be noted that although the image enhancement device 900 described above only shows a memory, processor, and communication interface, those skilled in the art should understand that in specific implementations, the image enhancement device 900 may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the image enhancement device 900 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the image enhancement device 900 may only include the devices necessary for implementing the embodiments of this application, and may not necessarily include... Figure 22All the devices shown.

[0349] This application also provides a chip, which includes a transceiver unit and a processing unit. The transceiver unit may be an input / output circuit or a communication interface; the processing unit is a processor, microprocessor, or integrated circuit integrated on the chip. This chip can execute the image enhancement method described in the above method embodiments.

[0350] This application also provides a computer-readable storage medium storing instructions thereon, which, when executed, perform the image enhancement method described in the above method embodiments.

[0351] This application also provides a computer program product containing instructions that, when executed, perform the image enhancement method described in the above method embodiments.

[0352] It should also be understood that, in embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

[0353] It should also be understood that, in embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor. A portion of the processor may also include non-volatile random access memory. For example, the processor may also store device type information.

[0354] It should be understood that the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0355] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0356] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0357] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0358] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0359] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0360] In addition, the functional modules in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0361] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0362] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image enhancement method, characterized in that, include: A first high dynamic range (HDR) image corresponding to the image to be processed and the color image features of the image to be processed are obtained. The first HDR image is obtained by downsampling and HDR enhancement of the image to be processed. The color image features are used to indicate different brightness areas or different color change areas in the image to be processed. The image to be processed is an image with a first resolution. The first HDR image is an image with a second resolution. The first resolution is greater than the second resolution. The first HDR image is input into the first neural network unit for super-resolution processing; The first HDR image after super-resolution processing and the color image features are subjected to image enhancement processing by the second neural network unit to obtain the first image corresponding to the image to be processed. The first image refers to an image with a resolution of the first resolution. The first neural network unit and the second neural network unit are different modules in the same neural network model.

2. The image enhancement method as described in claim 1, characterized in that, Also includes: Obtain the texture image features of the image to be processed, wherein the texture image features are used to indicate the edge region or texture region of the image to be processed; The step of inputting the first HDR image into the first neural network unit for super-resolution processing includes: The first neural network unit performs super-resolution processing on the first HDR image based on the texture image features.

3. The image enhancement method as described in claim 1, characterized in that, Also includes: Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different; The step of inputting the first HDR image into the first neural network unit for super-resolution processing includes: The first neural network unit performs super-resolution processing on the first HDR image based on the multi-scale image features.

4. The image enhancement method as described in claim 1, characterized in that, Also includes: Obtain the texture image features of the image to be processed, wherein the texture image features are used to indicate the edge region or texture region of the image to be processed; Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different; The step of inputting the first HDR image into the first neural network unit for super-resolution processing includes: The first neural network unit performs super-resolution processing on the first HDR image based on the texture image features and the multi-scale image features.

5. The image enhancement method as described in claim 4, characterized in that, The step of performing super-resolution processing on the first HDR image using the first neural network unit based on the texture image features and the multi-scale image features includes: Obtain texture image features at a first scale, wherein the texture image features at the first scale have the same scale as the first scale image features in the multi-scale image features; The first neural network unit performs a dot product operation between the first HDR image and the texture image features at the first scale to obtain the third image features. The first neural network unit performs channel combining, convolution, and upsampling operations on the third image features and the first scale image features.

6. The image enhancement method according to any one of claims 1 to 5, characterized in that, The step of performing image enhancement processing on the first HDR image after super-resolution processing and the color image features through the second neural network unit to obtain the first image corresponding to the image to be processed includes: The first image is obtained by performing dot multiplication and convolution operations on the first HDR image after super-resolution processing and the color image features through the second neural network unit.

7. An image enhancement device, characterized in that, include: The acquisition unit is used to acquire a first high dynamic range (HDR) image corresponding to the image to be processed and the color image features of the image to be processed. The first HDR image is obtained by downsampling and HDR enhancement of the image to be processed. The color image features are used to indicate different brightness areas or different color change areas in the image to be processed. The image to be processed is an image with a first resolution. The first HDR image is an image with a second resolution. The first resolution is greater than the second resolution. The processing unit is used to input the first HDR image into the first neural network unit for super-resolution processing; and to perform image enhancement processing on the first HDR image after super-resolution processing and the color image features through the second neural network unit to obtain the first image corresponding to the image to be processed, wherein the first image refers to an image with a resolution of the first resolution, and the first neural network unit and the second neural network unit are different modules in the same neural network model.

8. The image enhancement apparatus as claimed in claim 7, characterized in that, The acquisition unit is also used for: Obtain the texture image features of the image to be processed, wherein the texture image features are used to indicate the edge region or texture region of the image to be processed; The processing unit is specifically used for: The first neural network unit performs super-resolution processing on the first HDR image based on the texture image features.

9. The image enhancement apparatus as claimed in claim 7, characterized in that, The acquisition unit is also used for: Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different; The processing unit is specifically used for: The first neural network unit performs super-resolution processing on the first HDR image based on the multi-scale image features.

10. The image enhancement apparatus as claimed in claim 7, characterized in that, The acquisition unit is also used for: Obtain the texture image features of the image to be processed, wherein the texture image features are used to indicate the edge region or texture region of the image to be processed; Obtain multi-scale image features of the image to be processed, wherein the multi-scale image features are used to indicate image information of the image to be processed at different scales, and the scale of any one of the multi-scale image features is different; The processing unit is specifically used for: The first neural network unit performs super-resolution processing on the first HDR image based on the texture image features and the multi-scale image features.

11. The image enhancement apparatus as claimed in claim 10, characterized in that, The acquisition unit is specifically used for: Obtain texture image features at a first scale, wherein the texture image features at the first scale have the same scale as the first scale image features in the multi-scale image features; The processing unit is specifically used for: The first neural network unit performs a dot product operation between the first HDR image and the texture image features at the first scale to obtain the third image features. The first neural network unit performs channel combining, convolution, and upsampling operations on the third image features and the first scale image features.

12. The image enhancement apparatus according to any one of claims 7 to 11, characterized in that, The processing unit is specifically used for: The first image is obtained by performing dot multiplication and convolution operations on the first HDR image after super-resolution processing and the color image features through the second neural network unit.

13. An image enhancement device, characterized in that, include: Memory, used to store programs; A processor for executing a program stored in the memory, wherein when the processor executes the program stored in the memory, the processor is configured to perform the image enhancement method according to any one of claims 1 to 6.

14. A computer-readable medium, characterized in that, The computer-readable medium stores computer program code that, when executed on a computer, causes the computer to perform the image enhancement method as described in any one of claims 1 to 6.

Images