Image enhancement method and related device

Abstract

The embodiment of the present application discloses an image enhancement method, which is applied to the field of artificial intelligence, including: a terminal inputs a first image into a first network, obtains one or more sets of intermediate features extracted by the first network, and the one or more sets of intermediate features are related to the hue component of the enhanced image of the first image; the terminal outputs a second image through processing by a second network, and the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more sets of intermediate features. The method locates and separates the reflective area in the image based on the hue component that can fully characterize the difference between the reflective layer and the transmissive layer, and can effectively remove the reflection phenomenon in the image.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image enhancement method and related devices. Background Art

[0002] With the popularity of terminal devices such as smartphones and tablets, it has become very common for people to take photos and videos through terminal devices. Due to the limitations of the shooting scene, if the user takes a photo through glass or windows, the captured image will usually have undesirable reflections. This reflection phenomenon usually causes problems such as background scene distortion, occlusion or blur in the image, thereby reducing the image quality. The reduction in image quality will affect the user's visual experience and will also have a serious impact on subsequent computer vision tasks (such as object detection and semantic segmentation). Therefore, eliminating the reflection phenomenon in the image and restoring the original image content can greatly improve the imaging quality and help improve the accuracy of subsequent computer vision task processing.

[0003] At present, the de-reflection method in the related art assists de-reflection by manually designing prior information, such as manually designing gradient sparsity, out-of-focus blur characteristics, depth of field characteristics and other information. Since these manually designed prior information only consider the reflection characteristics in specific scenes, when de-reflection is performed based on these prior information, a good de-reflection effect can only be obtained in specific scenes. Once the reflective scene violates the established assumptions, these methods cannot correctly remove the reflection phenomenon in the image. Therefore, there is an urgent need for a method that can effectively remove the reflection phenomenon in the image. Summary of the invention

[0004] The present application provides an image enhancement method and a related device, which obtains intermediate features extracted by a first network, the intermediate features are related to the hue component of an enhanced image of an input image, and inputs the intermediate features into a second network for performing image enhancement, thereby introducing a hue component in the second network that can fully characterize the difference between a reflective layer and a transmissive layer, and can more accurately locate and separate the reflective area in the image during image enhancement, and effectively remove the reflection phenomenon in the image.

[0005] The first aspect of the present application provides an image enhancement method, comprising: a terminal inputs a first image into a first network to obtain one or more sets of intermediate features extracted by the first network. The first network may be a trained network model, and the first network is used to predict the hue image of an enhanced image of the first image. The hue image includes a hue component in a hue, saturation, and value (HSV) color space. The enhanced image of the first image is specifically an image obtained after image enhancement is performed on the first image. In this way, the one or more sets of intermediate features extracted by the first network are related to the hue component of the enhanced image of the first image. The hue component is a component indicating color information in the hue, saturation, and value HSV color space.

[0006] Then, the terminal outputs a second image through processing by the second network, the second image being an enhanced image of the first image, and the input of the second network including the first image and the one or more sets of intermediate features. Specifically, after obtaining the one or more sets of intermediate features extracted by the first network, the terminal uses the first image as the original input of the second network and inputs it into the second network; the one or more sets of intermediate features extracted by the first network are used as the intermediate input of the second network and input into the second network.

[0007] In this scheme, by obtaining the intermediate features extracted by the first network, the intermediate features are related to the hue component of the enhanced image of the input image, and the intermediate features are input into the second network for performing image enhancement, thereby introducing the hue component that can fully characterize the difference between the reflective layer and the transmissive layer into the second network, which can more accurately locate and separate the reflective area in the image during image enhancement, and effectively remove the reflection phenomenon in the image.

[0008] In a possible implementation, the terminal inputs the first image into the first network to obtain one or more sets of intermediate features extracted by the first network, including: the terminal extracts the one or more sets of intermediate features through one or more first convolutional layers in series in the first network. That is, the first network includes one or more first convolutional layers in series, and each first convolutional layer is used to extract a set of intermediate features to obtain multiple sets of intermediate features of different scales.

[0009] In a possible implementation, the first convolutional layer includes multiple layers of serially connected convolutional layers. The first convolutional layer may, for example, include multiple layers of serially connected atrous convolutional layers.

[0010] Since the tonal image includes significant reflection features and contains fewer details, the use of a serial atrous convolutional layer structure in the first convolutional layer can quickly improve the multi-scale expression capability of the hierarchical approach, which is beneficial for locating the reflection area in space.

[0011] In a possible implementation, the terminal outputs the second image through processing by the second network, specifically including: extracting features of the first image through the second network; fusing the features of the first image and the one or more groups of intermediate features through the second network to obtain one or more groups of fused features; and processing the one or more groups of fused features through the second network to obtain the second image.

[0012] In this scheme, by fusing the intermediate features extracted by the first network with the features of the first image extracted by the second network, the hue components that characterize the difference between the reflective layer and the transmissive layer can be effectively fused, and the second network's emphasis on significant reflective areas can be strengthened, providing a more accurate optimization direction for network training.

[0013] In a possible implementation, the fusing of the features of the first image and the one or more sets of intermediate features through the second network to obtain fused features includes: fusing the features of the first image and the one or more sets of intermediate features through one or more feature units in series in the second network to obtain one or more sets of fused features. Each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fused features output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is the corresponding set of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding set of intermediate features.

[0014] In this solution, by fusing one or more sets of intermediate features extracted by the first network with the features of one or more first images extracted by the second network, the hue components that characterize the difference between the reflective layer and the transmissive layer can be effectively fused, and the second network's emphasis on significant reflective areas can be strengthened, providing a more accurate optimization direction for network training.

[0015] In a possible implementation, the second convolutional layer includes multiple layers of parallel connected convolutional layers. The second convolutional layer may, for example, include multiple layers of parallel atrous convolutional layers.

[0016] In this scheme, the second convolutional layer extracts the features of the image by using a parallel hole convolutional layer, which can effectively restore the clean and detail-rich transmission layer in the image.

[0017] In a possible implementation, the second network may also include an attention module of a convolution block, the attention module of the convolution block is connected to the first feature unit in the second network, the attention module of the convolution block is used to extract features of the first image, and the first feature unit in the second network is used to perform feature extraction on the features extracted by the attention module of the convolution block. Simply put, in the second network, the attention module of the convolution block is located before the first second feature extraction module. After the attention module of the convolution block extracts features of the image input to the second network, the second feature extraction module further extracts features extracted by the attention module of the convolution block.

[0018] In this scheme, by continuously inferring the attention maps of spatial and channel information through the attention module of the convolution block in the second network, the features can be adaptively refined to improve the effect of image de-reflection.

[0019] In a possible implementation, the method also includes: the terminal processes the first image to obtain a hue image corresponding to the first image, and the hue image includes a hue component of an HSV color space; the terminal inputs the hue image corresponding to the first image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

[0020] In a possible implementation, the method further includes: the terminal inputs the first image into the first network to obtain a hue image of the first image; the hue image is input into the first network to be processed by the first network together with the first image to obtain the one or more sets of intermediate features, wherein the hue image includes a hue component of an HSV color space. That is, after the first image is input into the first network, the first network can first convert the first image into a hue image, and based on the first image and the hue image corresponding to the first image, process to obtain the one or more sets of intermediate features.

[0021] In a possible implementation, the input of the second network also includes semantic features of the first image. The semantic features may specifically refer to high-level features. High-level features are relative to low-level features, and low-level features refer to detailed features such as specific contours, edges, colors, textures, and shapes in an image. High-level features are features used to describe objects in an image and can be used to identify and classify objects in an image.

[0022] In this solution, by introducing semantic features, the semantic understanding ability of the second network can be improved, allowing the second network to ensure the image quality after de-reflection in a relatively high dimension, such as preventing color cast in the image and removing large reflective areas, ultimately improving the overall quality of the de-reflection result.

[0023] In a possible implementation, the method is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing, and image de-fogging.

[0024] The second aspect of the present application provides a training method for a network model, comprising: obtaining an image sample pair, the image sample pair comprising a first image and an enhanced image of the first image; inputting the first image into a first network to obtain one or more groups of intermediate features extracted by the first network, the one or more groups of intermediate features being related to the hue component of the enhanced image of the first image, the hue component being a component indicating color information in the hue, saturation, and brightness HSV color space; outputting a second image through processing by the second network, the second image being an enhanced image of the first image, the input of the second network comprising the first image and the one or more groups of intermediate features; obtaining a first loss function based on the enhanced image and the second image in the image sample pair, the first loss function being used to indicate the difference between the enhanced image and the second image in the image sample pair; training the first network and the second network based on the first loss function to obtain a trained first network and a trained second network.

[0025] In a possible implementation, the method further includes: obtaining a hue image of the second image and a hue image of an enhanced image in an image sample pair; obtaining a second loss function based on the hue image of the second image and the hue image of the enhanced image in the image sample pair, the second loss function being used to indicate the difference between the hue image of the second image and the hue image of the enhanced image in the image sample pair; and training the first network and the second network based on the first loss function includes: training the first network and the second network based on the first loss function and the second loss function.

[0026] In a possible implementation, the method also includes: obtaining a hue image of an enhanced image in an image sample pair and a third image output by the first network; obtaining a third loss function based on the hue image of the enhanced image in the image sample pair and the third image, wherein the third loss function is used to indicate the difference between the hue image of the enhanced image in the image sample pair and the third image to obtain the third loss function; training the first network and the second network based on the first loss function and the second loss function includes: training the first network and the second network based on the first loss function, the second loss function and the third loss function.

[0027] In a possible implementation, inputting the first image into the first network to obtain one or more sets of intermediate features extracted by the first network includes: extracting the one or more sets of intermediate features through one or more first convolutional layers in series in the first network.

[0028] In a possible implementation, the first convolutional layer includes multiple layers of serially connected convolutional layers.

[0029] In a possible implementation, outputting the second image through processing by the second network includes: extracting features of the first image through the second network; fusing the features of the first image and the one or more groups of intermediate features through the second network to obtain one or more groups of fused features; and processing the one or more groups of fused features through the second network to obtain the second image.

[0030] In a possible implementation, the fusing of the features of the first image and the one or more groups of intermediate features through the second network to obtain fused features includes: fusing the features of the first image and the one or more groups of intermediate features through one or more feature units in series in the second network to obtain one or more groups of fused features; wherein each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fused features output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is the corresponding group of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding group of intermediate features.

[0031] In a possible implementation manner, the second convolutional layer includes multiple layers of parallel connected convolutional layers.

[0032] In a possible implementation, the method further includes: processing the first image to obtain a hue image corresponding to the first image, the hue image including a hue component of an HSV color space; inputting the hue image corresponding to the first image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

[0033] In a possible implementation, the method further includes: inputting the first image into a first network to obtain a hue image of the first image; inputting the hue image into the first network to be processed by the first network together with the first image to obtain the one or more sets of intermediate features, wherein the hue image includes a hue component of an HSV color space.

[0034] In a possible implementation manner, the input of the second network also includes semantic features of the first image.

[0035] In a possible implementation, the method is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing, and image de-fogging.

[0036] The third aspect of the present application provides an image processing device, comprising: a processing unit. The processing unit is used to: input a first image into a first network to obtain one or more sets of intermediate features extracted by the first network, wherein the one or more sets of intermediate features are related to the hue component of an enhanced image of the first image, wherein the hue component is a component indicating color information in the hue, saturation, and brightness HSV color space; and output a second image through processing by the second network, wherein the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more sets of intermediate features.

[0037] In a possible implementation, the processing unit is further used to extract the one or more groups of intermediate features through one or more first convolutional layers in series in the first network.

[0038] In a possible implementation, the first convolutional layer includes multiple layers of serially connected convolutional layers.

[0039] In a possible implementation, the processing unit is further used to: extract features of the first image through the second network; fuse the features of the first image and the one or more groups of intermediate features through the second network to obtain one or more groups of fused features; and process the one or more groups of fused features through the second network to obtain the second image.

[0040] In a possible implementation, one or more groups of fused features are obtained by fusing the features of the first image and the one or more groups of intermediate features through one or more feature units in series in the second network; wherein each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fused features output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is a corresponding group of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding group of intermediate features.

[0041] In a possible implementation manner, the second convolutional layer includes multiple layers of parallel connected convolutional layers.

[0042] In a possible implementation, the processing unit is further used to: process the first image to obtain a hue image corresponding to the first image, wherein the hue image includes a hue component in an HSV color space; and input the hue image corresponding to the first image into the first network so as to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

[0043] In a possible implementation, the processing unit is further used to: input the first image into the first network to obtain a hue image of the first image; input the hue image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features, wherein the hue image includes a hue component of the HSV color space.

[0044] In a possible implementation manner, the input of the second network also includes semantic features of the first image.

[0045] In a possible implementation manner, the device is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing, and image de-fogging.

[0046] The fourth aspect of the present application provides a training device, comprising: an acquisition unit and a processing unit. The acquisition unit is used to acquire an image sample pair, wherein the image sample pair includes a first image and an enhanced image of the first image; the processing unit is used to: input the first image into a first network to obtain one or more sets of intermediate features extracted by the first network, wherein the one or more sets of intermediate features are related to the hue component of the enhanced image of the first image, wherein the hue component is a component indicating color information in the hue, saturation, and brightness HSV color space; output a second image through processing by the second network, wherein the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more sets of intermediate features; acquire a first loss function according to the enhanced image and the second image in the image sample pair, wherein the first loss function is used to indicate the difference between the enhanced image and the second image in the image sample pair; train the first network and the second network according to the first loss function to obtain a trained first network and a trained second network.

[0047] In a possible implementation, the acquisition unit is further used to acquire a hue image of the second image and a hue image of an enhanced image in an image sample pair; the acquisition unit is further used to acquire a second loss function based on the hue image of the second image and the hue image of the enhanced image in the image sample pair, wherein the second loss function is used to indicate the difference between the hue image of the second image and the hue image of the enhanced image in the image sample pair; the processing unit is further used to train the first network and the second network based on the first loss function and the second loss function.

[0048] In a possible implementation, the acquisition unit is further used to acquire a hue image of an enhanced image in the image sample pair and a third image output by the first network; the acquisition unit is further used to acquire a third loss function based on the hue image of the enhanced image in the image sample pair and the third image, and the third loss function is used to indicate the difference between the hue image of the enhanced image in the image sample pair and the third image to obtain the third loss function; the processing unit is further used to train the first network and the second network based on the first loss function, the second loss function and the third loss function.

[0049] In a possible implementation, the processing unit is further used to extract the one or more groups of intermediate features through one or more first convolutional layers in series in the first network.

[0050] In a possible implementation, the first convolutional layer includes multiple layers of serially connected convolutional layers.

[0051] In a possible implementation, the processing unit is further used to extract features of the first image through the second network; fuse the features of the first image and the one or more groups of intermediate features through the second network to obtain one or more groups of fused features; and process the one or more groups of fused features through the second network to obtain the second image.

[0052] In a possible implementation, the processing unit is also used to fuse the features of the first image and the one or more groups of intermediate features through one or more feature units in series in the second network to obtain one or more groups of fused features; wherein each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fused features output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is a corresponding group of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding group of intermediate features.

[0053] In a possible implementation manner, the second convolutional layer includes multiple layers of parallel connected convolutional layers.

[0054] In a possible implementation, the processing unit is also used to process the first image to obtain a hue image corresponding to the first image, wherein the hue image includes a hue component in an HSV color space; and input the hue image corresponding to the first image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

[0055] In a possible implementation, the processing unit is further used to input the first image into the first network to obtain a hue image of the first image; input the hue image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features, and the hue image includes a hue component of the HSV color space.

[0056] In a possible implementation manner, the input of the second network also includes semantic features of the first image.

[0057] In a possible implementation manner, the device is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing, and image de-fogging.

[0058] In a fifth aspect, the present application provides an image processing device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect or the second aspect is implemented. For the processor to execute the steps in each possible implementation of the first aspect or the second aspect, the details can be referred to the first aspect, and no further description is given here.

[0059] In a sixth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first or second aspect above.

[0060] A seventh aspect of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or the second aspect.

[0061] In an eighth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method described in the first or second aspect above.

[0062] In a ninth aspect of the present application, a chip system is provided, which includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in the first or second aspect above, for example, sending or processing the data and / or information involved in the above method. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of chips, or it can include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A structural diagram of the main framework of artificial intelligence;

[0064] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0065] Figure 3 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;

[0066] Figure 4 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0067] Figure 5a A schematic diagram of a comparison of an image before and after removing reflection provided in an embodiment of the present application;

[0068] Figure 5b A schematic diagram of comparison of an image before and after shadow removal provided in an embodiment of the present application;

[0069] Figure 6 A schematic diagram of a flow chart of an image enhancement method provided in an embodiment of the present application;

[0070] Figure 7 A schematic diagram of a HSV color space model provided in an embodiment of the present application;

[0071] Figure 8a A schematic diagram of a comparison between an RGB image without reflection phenomenon and a hue image provided in an embodiment of the present application;

[0072] Figure 8b A schematic diagram of a comparison between an RGB image with a reflection phenomenon and a hue image provided in an embodiment of the present application;

[0073] Figure 8c A schematic diagram of the structure of a first network and a second network provided in an embodiment of the present application;

[0074] Figure 9a A schematic diagram of the structure of a first feature extraction module provided in an embodiment of the present application;

[0075] Figure 9b A schematic diagram of the structure of a second feature extraction module provided in an embodiment of the present application;

[0076] Fig.9c A structural diagram of a feature fusion module provided in an embodiment of the present application;

[0077] Fig.10 A schematic diagram of the structure of an attention module of a convolutional block provided in an embodiment of the present application;

[0078] Fig.11a A schematic diagram of the structure of a first network provided in an embodiment of the present application;

[0079] Fig.11b A schematic diagram of the structure of a second network provided in an embodiment of the present application;

[0080] Fig.12 A schematic diagram of the structure of an image de-reflection network provided in an embodiment of the present application;

[0081] Fig.13 A schematic diagram comparing the input and output of an image de-reflection network provided in an embodiment of the present application;

[0082] Fig.14 A flowchart of a network model training method provided in an embodiment of the present application;

[0083] Fig.15 A schematic diagram of a distance measurement of a hue image provided in an embodiment of the present application;

[0084] Fig.16 A schematic diagram comparing the network de-reflection effects provided in the embodiments of the present application;

[0085] Fig.17 A schematic diagram comparing the de-reflection effects of introducing different networks provided in an embodiment of the present application;

[0086] Fig.18 A schematic diagram comparing the de-reflection effects of different image de-reflection methods provided in the embodiments of the present application;

[0087] Fig.19 A comparative schematic diagram of an image de-reflection effect provided in an embodiment of the present application;

[0088] Fig. 20 A schematic diagram for comparing another image de-reflection effect provided by an embodiment of the present application;

[0089] Fig.21 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0090] Fig. 22A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0091] Fig.23 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0092] Fig.24 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0093] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0094] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0095] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0096] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0097] (1) Infrastructure.

[0098] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0099] (2)Data.

[0100] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0101] (3) Data processing

[0102] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0103] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0104] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0105] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0106] (4) General ability.

[0107] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0108] (5) Smart products and industry applications.

[0109] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, smart cities, etc.

[0110] The following describes the method provided by this application from the perspectives of model training and model application:

[0111] The model training method provided in the embodiment of the present application can be specifically applied to data training, machine learning, deep learning and other data processing methods to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, training, etc. on the training data, and finally obtain a trained neural network model (such as the target neural network model in the embodiment of the present application); and the target neural network model can be used for model reasoning, specifically, the input data can be input into the target neural network model to obtain output data.

[0112] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0113] (1) Neural network.

[0114] A neural network may be composed of neural units, and a neural unit may refer to an operation unit that takes xs (i.e., input data) and intercept 1 as input, and the output of the operation unit may be:

[0115] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0116] (2) Convolutional neural network (CNN) is a deep neural network with a convolutional structure. The convolutional neural network includes a feature extractor consisting of a convolution layer and a subsampling layer. The feature extractor can be regarded as a filter, and the convolution process can be regarded as using a trainable filter to convolve with an input image or convolution feature plane (feature map). The convolution layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal (for example, the first convolution layer and the second convolution layer in this embodiment). In the convolution layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, and each feature plane can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are the convolution kernels. Shared weights can be understood as the way of extracting image information is independent of position. The implicit principle is that the statistical information of a part of the image is the same as that of other parts. This means that the image information learned in one part can also be used in another part. So for all positions on the image, we can use the same learned image information. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally speaking, the more convolution kernels there are, the richer the image information reflected by the convolution operation.

[0117] The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connections between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0118] Specifically, Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / pooling layer 120 , wherein the pooling layer is optional, and a neural network layer 130 .

[0119] Among them, the structure composed of the convolution layer / pooling layer 120 and the neural network layer 130 can be the first convolution layer and the second convolution layer described in the present application, the input layer 110 is connected to the convolution layer / pooling layer 120, the convolution layer / pooling layer 120 is connected to the neural network layer 130, the output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform nonlinear processing on the output of the neural network layer 130.

[0120] Convolutional layer / pooling layer 120. Convolutional layer: Figure 2The convolution layer / pooling layer 120 shown may include layers 121-126 as shown in the examples. In one implementation, layer 121 is a convolution layer, layer 122 is a pooling layer, layer 123 is a convolution layer, layer 124 is a pooling layer, layer 125 is a convolution layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolution layers, layer 123 is a pooling layer, layers 124 and 125 are convolution layers, and layer 126 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0121] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.

[0122] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0123] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers (for example, 126) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0124] Pooling layer: Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer, such as Figure 2 The layers 121-126 shown in 120 may be a convolution layer followed by a pooling layer, or multiple convolution layers may be followed by one or more pooling layers.

[0125] Neural network layer 130: After being processed by the convolution layer / pooling layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 2 131, 132 to 13n) and the output layer 140 shown, the parameters contained in the multi-layer hidden layers can be pre-trained according to relevant training data of a specific task type. For example, the task type may include image recognition, image classification, image super-resolution reconstruction, etc.

[0126] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 (such as Figure 2 The propagation from 110 to 140 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 140 to 110 is called back propagation) and then the weight values ​​and biases of the aforementioned layers will begin to be updated to reduce the loss of the convolutional neural network 100 and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result.

[0127] It should be noted that if Figure 2 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, Figure 3The multiple convolutional layers / pooling layers shown are operated in parallel, and the features extracted respectively are input to the full neural network layer 130 for processing.

[0128] (3) Deep neural networks.

[0129] Deep Neural Network (DNN), also known as multi-layer neural network, can be understood as a neural network with many hidden layers. There is no special metric for "many" here. From the position of different layers of DNN, the neural network inside DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer. Although DNN looks complicated, the work of each layer is actually not complicated. Simply put, it is the following linear relationship expression: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The definition of these parameters in DNN is as follows: Take coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0130] (4) Loss function

[0131] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0132] (5) Back propagation algorithm.

[0133] Convolutional neural networks can use the error back propagation algorithm to correct the size of the parameters in the initial super-resolution model during the training process, so that the reconstruction error loss of the super-resolution model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the error loss information is back-propagated to update the parameters in the initial super-resolution model, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the super-resolution model, such as the weight matrix.

[0134] (6) Linear operation.

[0135] Linearity refers to the proportional, linear relationship between quantities. Mathematically, it can be understood as a function whose first-order derivative is a constant. Linear operations can be, but are not limited to, addition operations, null operations, identity operations, convolution operations, batch normalization (BN) operations, and pooling operations. Linear operations can also be called linear mappings. Linear mappings need to meet two conditions: homogeneity and additivity. If either condition is not met, it is nonlinear.

[0136] Among them, homogeneity means f(ax)=af(x); additivity means f(x+y)=f(x)+f(y); for example, f(x)=ax is linear. It should be noted that x, a, and f(x) here are not necessarily scalars, but can be vectors or matrices to form a linear space of any dimension. If x and f(x) are n-dimensional vectors, when a is a constant, they are equivalent to satisfying homogeneity, and when a is a matrix, they are equivalent to satisfying additivity. Relatively speaking, a function graph that is a straight line does not necessarily conform to a linear mapping. For example, f(x)=ax+b neither satisfies homogeneity nor additivity, so it belongs to a nonlinear mapping.

[0137] In the embodiment of the present application, a combination of multiple linear operations may be referred to as a linear operation, and each linear operation included in a linear operation may also be referred to as a sub-linear operation.

[0138] Figure 4 is a schematic diagram of a processing system 400 architecture provided in an embodiment of the present application. Figure 4 In the embodiment, the execution device 410 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user can input data to the I / O interface 112 through a client device 440 .

[0139] When the execution device 410 preprocesses the input data, or when the computing module 111 of the execution device 410 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 410 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0140] Finally, the I / O interface 112 returns the processing result to the client device 440 so as to provide it to the user.

[0141] Optionally, the client device 440 may be, for example, a control unit in an autonomous driving system or a functional algorithm module in a mobile phone terminal. For example, the functional algorithm module may be used to implement related tasks.

[0142] It is worth noting that the training device 420 can generate corresponding target models / rules (such as the target neural network model in this embodiment) based on different training data for different goals or different tasks. The corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0143] exist Figure 4In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 440 can automatically send input data to the I / O interface 112. If the client device 440 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 440. The user can view the results output by the execution device 410 on the client device 440, and the specific form can be display, sound, action and other specific methods. The client device 440 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data, and store them in the database 430. Of course, it is also possible not to collect through the client device 440, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 430.

[0144] It is worth noting that Figure 4 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 4 In the embodiment, the data storage system 150 is an external memory relative to the execution device 410. In other cases, the data storage system 150 may also be placed in the execution device 410.

[0145] In order to remove reflections in images, traditional de-reflection methods use manually designed prior information to assist de-reflection, such as manually designed gradient sparsity, out-of-focus blur characteristics, depth of field characteristics, etc. Since these manually designed prior information only considers the reflection characteristics in specific scenes, when de-reflection is performed based on these prior information, good de-reflection effects can only be obtained in specific scenes. Once the reflective scene violates the established assumptions, these methods cannot correctly remove the reflection phenomenon in the image.

[0146] For image de-reflection tasks, the widely used imaging model is: reflection distorted image I = transmission layer T + reflection layer R, that is, the reflection distorted image I is a linear combination of the transmission layer T and the reflection layer R. De-reflection is to separate the reflection layer R from the reflection distorted image I to obtain the desired transmission layer T. Since the transmission layer T and the reflection layer R are both unknown variables, additional auxiliary information is needed to separate the reflection layer R. The reflection layers of real scenes come from various sources and have complex characteristics, so it is difficult to find an effective auxiliary information that can achieve effective reflection removal.

[0147] In recent years, with the progress of deep learning and computer vision, convolutional neural networks have become the standard solution for most visual problems, including image de-reflection. Convolutional neural networks are based on a large number of trainable convolution kernels, and the parameters of these convolution kernels are optimized through the supervision of specific loss functions. Neural network-based methods can explore different auxiliary information, such as underlying edge information, high-level semantic information, etc. These auxiliary information combined with training data help the network learn de-reflection, and ultimately the de-reflection performance has been greatly improved. However, these auxiliary information are limited by their own characteristics and cannot fully characterize the characteristics of the reflection layer and the transmission layer. It is still difficult to effectively remove reflections in some complex reflection scenes. For example, edge information, as a sparse feature of the image, only considers the difference between adjacent pixels and cannot describe the reflection of a larger spatial range, so the reflection performance of the highlighted area is poor.

[0148] In view of this, an image enhancement method provided in an embodiment of the present application introduces the image tone information as auxiliary information, which can fully characterize the difference between the reflective layer and the transmissive layer, help locate and separate the reflective area in the image, and thus effectively remove the reflection phenomenon in the image.

[0149] In addition, the image enhancement method provided in the embodiment of the present application can be used to remove the reflection phenomenon in the image, and can also be used to remove the shadow in the image or remove the fog in the image. In short, when there is reflection, shadow or fog in the image, the image enhancement method provided in the embodiment of the present application can be used to achieve image enhancement, thereby removing the reflection layer, shadow layer or fog layer in the image. For the sake of ease of description, the image enhancement method provided in the embodiment of the present application will be described in detail below, taking image dereflection as an example.

[0150] For example, see Figure 5a , Figure 5a This is a schematic diagram of a comparison of an image before and after removing reflection provided by an embodiment of the present application. Figure 5a As shown in the figure, before image de-reflection, there is obvious reflection in the left area of ​​the image, and the reflection layer and the transmission layer in the image overlap, resulting in occlusion of the image. After image de-reflection, the reflection in the left area of ​​the image is eliminated, and the reflection layer in the image is removed, so the problem of occlusion of the transmission layer in the image is well solved.

[0151] See also Figure 5b , Figure 5b This is a schematic diagram of a comparison of an image before and after removing shadows provided by an embodiment of the present application. Figure 5bAs shown in the figure, the image shows the ground under the sun. Before the image is deshadowed, there is an obvious shadow in the upper area of ​​the image, which is specifically the shadow of a person holding an umbrella under the sun. In the image, the shadow blocks the tiles on the ground, resulting in an obvious dark layer in the blocked part of the ground. After the image is deshadowed, the shadow phenomenon in the upper area of ​​the image is eliminated, and the shadow layer in the image is removed, so the problem of the ground being blocked in the image is well solved.

[0152] The image enhancement method provided in the embodiment of the present application can be applied to a terminal or a server. Exemplarily, the terminal can be, for example, a digital camera, a surveillance camera, a mobile phone, a personal computer (PC), a laptop, a server, a tablet computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. For the sake of convenience, the image enhancement method provided in the embodiment of the present application will be introduced below by taking the application of the image enhancement method to the terminal as an example.

[0153] See also Figure 6 , Figure 6 The following is a flow chart of an image enhancement method provided in an embodiment of the present application. Figure 6 As shown, the image enhancement method includes the following steps 601-602.

[0154] Step 601: input a first image into a first network to obtain one or more sets of intermediate features extracted by the first network.

[0155] In this embodiment, the one or more groups of intermediate features are related to the hue component of the enhanced image of the first image, and the hue component is a component indicating color information in the HSV color space. Optionally, the first network can be a trained network model, and the first network is used to predict the hue image of the enhanced image of the first image. The hue image includes the hue component of the HSV color space. The enhanced image of the first image is specifically an image obtained after performing image enhancement on the first image. In this way, the one or more groups of intermediate features extracted by the first network are related to the hue component of the enhanced image of the first image. The hue component is a component indicating color information in the HSV color space of hue, saturation, and brightness.

[0156] Exemplarily, performing image enhancement on an image may refer to performing de-reflection, de-shadowing or de-fogging operations on the image. That is, when the image enhancement method provided in this embodiment is used to perform image de-reflection, the enhanced image of the first image may refer to the image obtained after performing image de-reflection on the first image.

[0157] The hue image of the enhanced image may include only the hue component of the HSV color space. The HSV color space is a color space created based on the intuitive characteristics of color, also known as the Hexcone Model. Figure 7 , Figure 7 A schematic diagram of a model of an HSV color space provided for an embodiment of the present application. In the HSV color space, the parameters of color are: hue (H), saturation (S), and lightness (V). Hue can be specifically measured by angle, with a value range of 0° to 360°, starting from red and calculated in a counterclockwise direction, with red being 0°, green being 120°, and blue being 240°. Their complementary colors are: yellow is 60°, cyan is 180°, and purple is 300°. Simply put, the hue parameter represents color information, that is, the position of the spectral color.

[0158] Optionally, the first image in this embodiment can specifically be an RGB image. An RGB image refers to an image represented by three channels, which are red, green, and blue. Different combinations of these three colors can form almost all other colors, so an RGB image is actually an image obtained by combining these three colors. Generally speaking, RGB images and HSV images can be converted to each other. For example, by performing color space conversion on an RGB image, a corresponding HSV image can be obtained; by performing color space conversion on an HSV image, a corresponding RGB image can be obtained.

[0159] For the hue image of the above-mentioned enhanced image, when the enhanced image is an RGB image, the hue image of the enhanced image can be understood as performing color space conversion on the enhanced image to obtain the HSV image corresponding to the enhanced image, and then only retaining the hue component in the HSV image corresponding to the enhanced image, thereby obtaining the hue image of the enhanced image.

[0160] Optionally, the first network includes one or more first feature extraction modules, and one or more groups of intermediate features extracted by the first network can be extracted by one or more first feature extraction modules. Each of the one or more first feature extraction modules includes a first convolutional layer, that is, the first network includes one or more first convolutional layers in series. Simply put, on the basis of the first network being used to predict the hue image of the enhanced image corresponding to the input image, the first feature extraction module in the first network is used to extract hue features, so that subsequent modules in the first network can predict the hue image of the enhanced image based on the hue features extracted by the first feature extraction module.

[0161] Optionally, in order to ensure the prediction accuracy of the first network, the first network may extract intermediate features based on the first image and a hue image corresponding to the first image, thereby obtaining a hue image of an enhanced image of the first image.

[0162] Exemplarily, there may be multiple ways to obtain the tone image corresponding to the first image.

[0163] In a possible implementation, before inputting the first image into the first network, the terminal may perform color space conversion on the first image to obtain a hue image corresponding to the first image. Exemplarily, the terminal may execute a color space conversion function (e.g., RGB>HSV conversion function) to convert and obtain a hue image corresponding to the first image. After obtaining the hue image corresponding to the first image, the hue image corresponding to the first image is input into the first network to be processed by the first network together with the first image to obtain the one or more sets of intermediate features.

[0164] In another possible implementation, the first network includes a color space conversion module, and the color space conversion module is used to convert the first image into a hue image. That is, after the first image is input into the first network, the first network can convert the first image into a hue image based on the color space conversion module, and process the first image and the hue image corresponding to the first image to obtain the one or more sets of intermediate features.

[0165] Step 602: output a second image through processing by the second network, where the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more groups of intermediate features.

[0166] After obtaining the intermediate features extracted by the first network, the terminal uses the first image as the original input of the second network and inputs it into the second network; the intermediate features extracted by the first network are used as the intermediate input of the second network and input into the second network. The second network is used to extract the features of the input first image and fuse the extracted features with the input intermediate features, thereby predicting an enhanced image of the first image, i.e., the second image.

[0167] Specifically, the terminal extracts features of the first image through the second network, and fuses the features of the first image and the one or more sets of intermediate features through the second network to obtain one or more sets of fused features. Finally, the one or more sets of fused features are processed through the second network to obtain the second image.

[0168] See also Figure 8a , Figure 8a A schematic diagram of a comparison between an RGB image without reflection phenomenon and a hue image provided in an embodiment of the present application. Figure 8a As shown, when there is no reflection phenomenon in the RGB image, the hue image corresponding to the RGB image can well reflect the outlines of the main objects in the RGB image, such as the outlines of the duck and the cradle in the RGB image.

[0169] See also Figure 8b , Figure 8b The following is a schematic diagram of a comparison between an RGB image with a reflection phenomenon and a hue image provided in an embodiment of the present application. Figure 8b As shown in the figure, when there is a reflection phenomenon in the RGB image, the hue image corresponding to the RGB image cannot reflect the outline of the main object in the RGB image due to the influence of the reflection area in the RGB image. For example, the hue image cannot well present the outline of the duck and the cradle in the RGB image. In other words, the hue image corresponding to the RGB image includes corresponding hue feature information, and this part of the hue feature information is associated with the reflection area in the RGB image. Based on the hue feature information in the hue image, the reflection area in the RGB image can be effectively located.

[0170] Therefore, in this embodiment, by obtaining the hue features extracted by the network for predicting the hue image, and inputting the hue features as auxiliary information into the network for predicting the enhanced image, the reflective area in the image is located and separated based on the hue features that can fully characterize the difference between the reflective layer and the transmissive layer, thereby effectively removing the reflection phenomenon in the image.

[0171] Optionally, the second network includes one or more feature units in series, each feature unit includes a second feature extraction module and a feature fusion module. Each second feature extraction module may include a second convolutional layer. The input of the second convolutional layer in each feature unit is the fusion feature output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is a corresponding set of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion modules in the one or more feature units have a one-to-one corresponding set of intermediate features.

[0172] The number of the first feature extraction module in the first network and the number of feature units in the second network may be the same. For example, the first network includes n first feature extraction modules, and the second network includes n second feature extraction modules and n feature fusion modules.

[0173] See also Figure 8c , Figure 8c The following is a schematic diagram of the structure of a first network and a second network provided in an embodiment of the present application. Figure 8c As shown, the first network includes feature extraction module A1, feature extraction module A2...feature extraction module An, a total of n feature extraction modules; the second network includes feature extraction module B1, feature extraction module B2...feature extraction module Bn, a total of n feature extraction modules; the second network includes feature fusion module 1, feature fusion module 2...feature fusion module n, a total of n feature fusion modules.

[0174] The n feature extraction modules in the first network are connected in sequence to extract intermediate feature 1, intermediate feature 2, ... intermediate feature n. Specifically, the feature extraction module A1 in the first network extracts features from the input data to obtain intermediate feature 1; the feature extraction module A2 extracts features from intermediate feature 2 to obtain intermediate feature 2, and so on. The feature extraction modules in the first network extract n intermediate features respectively.

[0175] The feature extraction module and feature fusion module in the second network constitute a feature unit, and n groups of feature units are connected in sequence. The input of the first feature extraction module in the second network is the input data of the second network, and the input data of other feature extraction modules of the second network are the features output by the previous feature fusion module; the input of each feature fusion module in the second network comes from the feature extraction module in the second network and the feature extraction module in the first network, respectively, and is used to fuse the features extracted by the feature extraction module in the second network and the features extracted by the feature extraction module in the first network to obtain fused features.

[0176] Specifically, the feature fusion module 1, feature fusion module 2, ..., feature fusion module n in the second network are respectively connected to the feature extraction module B1, feature extraction module B2, ..., feature extraction module Bn in the second network. The feature extraction module B1 extracts features from the input data in the second network, and the feature fusion module 1 performs feature fusion on the features extracted by the feature extraction module B1 and the intermediate feature 1 extracted by the feature extraction module A1 to obtain fused feature 1; the feature extraction module B2 extracts features from the fused feature 1, and the feature fusion module 2 performs feature fusion on the features extracted by the feature extraction module B2 and the intermediate feature 2 extracted by the feature extraction module A2 to obtain fused feature 2... By analogy, the feature fusion module in the second network performs feature fusion based on the intermediate features extracted by the feature extraction module in the first network and the features extracted by the feature extraction module in the second network, and the second network finally performs image prediction based on the features fused by the feature fusion module to predict an enhanced image of the first image.

[0177] Optionally, the first convolutional layer in the first network includes multiple layers of serially connected convolutional layers, and the second convolutional layer in the second network includes multiple layers of parallelly connected convolutional layers. Exemplarily, the first convolutional layer may include 4 serial layers of dilated convolutional layers, and the strides of the 4 dilated convolutional layers are {2, 4, 8, 4} respectively. It can be understood that in addition to the 4 layers of dilated convolutional layers as an example, the convolutional layers included in the first convolutional layer can also be other numbers of layers, for example, the first convolutional layer includes 6 or 8 serial layers of convolutional layers, and this embodiment does not specifically limit the number of convolutional layers included in the first convolutional layer. In addition, the dilated convolutional layer in the first convolutional layer can also adopt other strides, and this embodiment also does not specifically limit the stride of the convolutional layer included in the first convolutional layer.

[0178] Specifically, see Figure 9a , Figure 9a A schematic diagram of the structure of a first convolutional layer provided in an embodiment of the present application. Figure 9a As shown in the figure, the first convolutional layer consists of 4 layers of hole convolutional layers connected in sequence. Figure 9a D2, D4, D8, and D4 in the figure respectively indicate that the stride of the first atrous convolution layer is 2, the stride of the second atrous convolution layer is 4, the stride of the third atrous convolution layer is 8, and the stride of the fourth atrous convolution layer is 4.

[0179] Since the tonal image includes significant reflection features and fewer details, using this serial hole convolution layer structure in the first convolution layer can quickly improve the multi-scale expression capability of the layered approach, which is conducive to locating the reflection area in space.

[0180] Exemplarily, the second convolutional layer may include 3 layers of parallel dilated convolutional layers, and the step sizes of the 3 layers of dilated convolutional layers are {4, 8, 16} respectively. It is understandable that, in addition to the 3 layers of dilated convolutional layers used as an example, the convolutional layers included in the second feature module may also have other numbers of layers and other step sizes. For example, the second feature module includes 4 or 6 layers of parallel convolutional layers. This embodiment does not specifically limit the number of layers and the step sizes of the convolutional layers included in the first feature module. By using parallel dilated convolutional layers to form the second convolutional layer, the clean and detail-rich transmission layer in the image can be effectively restored.

[0181] Specifically, see Figure 9b , Figure 9b A schematic diagram of the structure of a second convolutional layer provided in an embodiment of the present application. Figure 9b As shown in the figure, the second convolutional layer is composed of three parallel atrous convolutional layers, and one atrous convolutional layer is connected before and after the second convolutional layer. The three atrous convolutional layers included in the second convolutional layer are represented as D4, D8, and D16, respectively, to indicate that the step sizes of the three atrous convolutional layers are {4, 8, 16} respectively; there is also a convolutional layer connected before the second convolutional layer, which is respectively connected to the three parallel atrous convolutional layers in the second convolutional layer; there is also a convolutional layer with a step size of 2 (i.e., D2 in the figure) connected before the second convolutional layer, which is respectively connected to the three parallel atrous convolutional layers in the second convolutional layer. The three parallel atrous convolutional layers in the second convolutional layer are also connected to a splicing module, which is used to splice the features extracted by the three parallel atrous convolutional layers, and the spliced ​​features are then processed by a convolutional layer with a step size of 2.

[0182] In this embodiment, by using multiple layers of serially connected convolutional layers to form the first convolutional layer and using multiple layers of parallel connected convolutional layers to form the second convolutional layer, the multi-scale representation capability of the overall network can be improved at a finer level, thereby effectively improving the performance of the network, that is, improving the de-reflection capability of the network.

[0183] Exemplarily, the feature fusion module in the second network can be used to perform feature concatenation operation on the intermediate features extracted by the first convolution layer and the features extracted by the second convolution layer, so as to obtain concatenated features, which are fused features. In addition, the feature fusion module can also add a Strengthen-Operate-Subtract (SOS) enhancement strategy to enhance the fusion operation, so as to effectively utilize two different feature information and obtain fused features with better fusion performance. Exemplarily, please refer to Fig.9c , Fig.9c This is a schematic diagram of the structure of a feature fusion module provided in an embodiment of the present application. Fig.9c As shown in the figure, after feature 1 and feature 2 are enhanced and fused by the splicing module, the fused features are processed by a convolution layer with a step size of 2. The features processed by the convolution layer are then reduced with feature 2 to finally obtain the fused features.

[0184] Optionally, the input of the second network also includes the semantic features of the first image. The terminal inputs the first image, the semantic features of the first image and the intermediate features into the second network to obtain a second image output by the second network. In other words, the terminal can use the first image and the semantic features of the first image as inputs to the second network, so that the second network performs image enhancement based on the first image and the semantic features of the first image.

[0185] Generally speaking, the semantics of an image are divided into low-level features and high-level features. The bottom-level features of an image refer to detailed features such as contours, edges, colors, textures, and shape features. Generally, the semantic information of the low-level features of an image is relatively small, but the target position is accurate. The high-level features of an image usually refer to what humans can see. For example, when extracting low-level features from a face, feature information such as connected contours, noses, and eyes can be extracted, and then the high-level features are displayed as a face. Simply put, high-level features are used to describe the features of objects in an image and can be used to identify and classify objects in an image. The semantic information of high-level features is usually richer, but the target position is relatively rough. In this embodiment, by introducing semantic features, the semantic understanding ability of the second network can be improved, so that the second network can ensure the image quality after de-reflection in a relatively high dimension, such as no color cast in the image color and removal of large reflective areas, and ultimately improve the overall quality of the de-reflection result.

[0186] In this embodiment, the terminal can extract the semantic features of the first image through a pre-trained network, for example, through a Visual Geometry Group Network (VGG). In practical applications, the terminal can also use other methods to extract the semantic features of the first image, and this embodiment does not specifically limit the method of extracting semantic features.

[0187] Optionally, the second network also includes a convolutional block attention module (CBAM), which is connected to the first feature unit in the second network, and is used to extract features of the first image. The first feature unit in the second network is used to extract features from the features extracted by the convolutional block attention module. In simple terms, in the second network, the convolutional block attention module is located before the first second feature extraction module. After the convolutional block attention module extracts features from the image input to the second network, the second feature extraction module further extracts features from the features extracted by the convolutional block attention module.

[0188] See also Fig.10 , Fig.10 A schematic diagram of the structure of an attention module of a convolutional block provided in an embodiment of the present application. Fig.10 As shown, the attention module of the convolution block includes a channel attention module and a spatial attention module connected in sequence. The attention module of the convolution block is actually a feedforward convolutional neural network of a simple and effective attention module. By giving an intermediate feature map, the attention module of the convolution block can infer the attention map in two independent dimensions, channels and spatial order, and then multiply the attention map into an input feature map for adaptive feature refinement. Since the attention module of the convolution block is a lightweight and universal module, the attention module of the convolution block can be seamlessly integrated into the convolutional neural network and only adds negligible computational overhead. In this embodiment, by using the attention module of the convolution block in the second network to continuously infer the attention map of the spatial domain and channel information, the features can be adaptively refined to improve the effect of image de-reflection.

[0189] The above describes the image enhancement method provided by the embodiment. The following describes the process of performing image dereflection using the image enhancement method provided by the embodiment in conjunction with a specific example.

[0190] See also Fig.11a , Fig.11a This is a schematic diagram of the structure of a first network provided in an embodiment of the present application. Fig.11aAs shown, the input of the first network is a reflection distortion image and a hue image of the reflection distortion image, and multiple feature modules in the first network extract different intermediate features respectively, and the output of the first network is a hue image of the reflection distortion image after de-reflection. The first network extracts significant reflection features as intermediate features by predicting the hue image of the reflection distortion image, and provides them to the second network to help the second network locate and remove the reflection area, thereby effectively removing the reflection area in the image and obtaining a high-quality restored image.

[0191] See also Fig.11b , Fig.11b This is a schematic diagram of the structure of a second network provided in an embodiment of the present application. Fig.11b As shown, the input of the second network is the reflection distortion map and the semantic features corresponding to the reflection distortion map, and the output of the second network is the de-reflected image corresponding to the reflection distortion map. Specifically, the input part of the second network introduces semantic features to improve the semantic understanding ability of the network, and then uses a lightweight convolutional block attention module to continuously infer the attention map of the spatial domain and channel information, so as to adaptively refine the features. The refined features are further extracted by the feature extraction module, and enhanced and fused with the intermediate features provided by the first network, and finally further refined by the residual module to obtain the final prediction result.

[0192] See also Fig.12 , Fig.12 This is a schematic diagram of the structure of an image de-reflection network provided in an embodiment of the present application. Fig.12 As shown, the image de-reflection network is composed of the first network and the second network. The feature extraction module in the first network and the feature extraction module and feature fusion module in the second network together constitute a basic unit. There are n basic units in the image de-reflection network.

[0193] Assume that the reflection distortion map is I, the hue image corresponding to the reflection distortion map is H, and the high-level semantic feature corresponding to the reflection distortion map is G. The intermediate features extracted by the first network are denoted as F1, F2...Fn, and the output of the first network is H'.

[0194] The first network takes the reflection distortion image I and the hue image H as input, and extracts features through n feature extraction modules. Specifically, the first network extracts feature F1 through the first feature extraction module and passes it to the corresponding feature fusion module in the second network; the first network extracts feature F2 through the second feature extraction module and passes it to the corresponding feature fusion module in the second network; and so on, until the first network extracts feature Fn through the nth feature extraction module and passes it to the last feature fusion module in the second network.

[0195] Assume that the features extracted by the feature extraction module in the second network are recorded as T1, T2...Tn, the features obtained after fusion by the feature fusion module are recorded as S1, S2...Sn, and the output of the second network is T'. Specifically, the second network extracts features through the attention module of the convolution block, and then extracts feature T1 through the first feature extraction module, and enhances and fuses feature T1 and feature F1 transmitted from the first network through feature fusion module 1 to obtain enhanced feature S1. The second network then extracts feature S1 through the second feature extraction module to obtain feature T2, and enhances and fuses feature T2 and feature F2 transmitted from the first network through feature fusion module 2 to obtain enhanced feature S2. Similarly, the second network finally extracts feature Sn-1 through the nth feature extraction module to obtain feature Tn, and enhances and fuses feature Tn and feature Fn transmitted from the first network through feature fusion module n to obtain enhanced feature Sn. Finally, the second network refines feature Sn through the residual module to obtain the final output T'.

[0196] See also Fig.13 , Fig.13 A schematic diagram showing the comparison of input and output of an image de-reflection network provided in an embodiment of the present application. Fig.13 As shown, the reflection distortion map and the hue image of the reflection distortion map are used as the input of the first network, and are input into the first network, and the output image of the first network is obtained. The reflection distortion map and the high-level semantic features corresponding to the reflection distortion map are used as the input of the second network, and are input into the second network; the intermediate features 1 and 2 extracted by the first network are also input into the second network, and finally the output image of the second network is obtained.

[0197] The above describes the image enhancement method provided by the embodiment of the present application, and the following describes a network model training method. Through the network model training method, the network for performing image enhancement described in the above embodiment can be trained.

[0198] See 14, Fig.14 A flowchart of a network model training method provided in an embodiment of the present application. Fig.14 As shown, a network model training method provided in an embodiment of the present application includes the following steps 1401-1405.

[0199] Step 1401: Acquire an image sample pair, where the image sample pair includes a first image and an enhanced image of the first image.

[0200] In this embodiment, before executing the training method of the network model, a set of image sample pairs can be obtained in advance, and the image sample pair set includes multiple image sample pairs. Each image sample pair includes a first image and an enhanced image corresponding to the first image. Exemplarily, the first image in the image sample pair can be a reflection distortion map, and the enhanced image of the first image can be a true value image, that is, a de-reflected image corresponding to the reflection distortion map.

[0201] There are multiple ways to obtain a set of image sample pairs.

[0202] In a possible implementation, different images of the same scene are captured under different circumstances, thereby obtaining an image sample pair. Taking the image sample pair as a reflection distorted image as an example, during the image capture process, an image under a normal scene can be captured as an enhanced image (i.e., a de-reflected image) corresponding to the first image, and an image under the same scene with a reflection phenomenon can be captured as the first image. For example, after the image under a static scene is captured as a de-reflected image through a terminal, the terminal position is kept unchanged, a transparent glass capable of reflecting light is placed in front of the terminal, and the image under the same scene is captured through the terminal as a reflection distorted image.

[0203] In another possible implementation, after acquiring an image without reflection phenomenon, noise can be added to the image through a specific algorithm to obtain a distorted image corresponding to the image. For example, when adding reflection noise to an image, the image before adding the reflection noise can be used as an enhanced image of the first image, and the image after adding the reflection noise can be used as the first image.

[0204] Step 1402: input the first image into the first network to obtain one or more sets of intermediate features extracted by the first network, wherein the one or more sets of intermediate features are related to the hue component of the enhanced image of the first image, and the hue component is a component indicating color information in the hue, saturation, and lightness HSV color space.

[0205] Step 1403: Output a second image through processing by the second network, where the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more groups of intermediate features.

[0206] In this embodiment, steps 1402-1403 are similar to the above-mentioned steps 601-602. For details, please refer to the above-mentioned steps 601-302, which will not be repeated here.

[0207] Step 1404: Acquire a first loss function according to the enhanced image and the second image in the image sample pair, where the first loss function is used to indicate the difference between the enhanced image and the second image in the image sample pair.

[0208] After obtaining the predicted image corresponding to the first image through the second network prediction, a first loss function can be calculated based on the enhanced image in the image sample pair and the obtained predicted image, wherein the first loss function is used to indicate the difference between the enhanced image in the image sample pair and the first predicted image.

[0209] Exemplarily, the first loss function may include one or more of a content loss function, a perceptual loss function, and an adversarial loss function. The content loss function is used to represent the content fidelity of the image. For example, the content loss function may be represented by an L1 paradigm. Specifically, the content loss function may be represented by Formula 1.

[0210]

[0211] Among them, L1 represents the content loss function, T represents the enhanced image in the image sample pair, Represents the predicted image obtained by the second network.

[0212] The perceptual loss function is used to provide supervisory constraints related to high-level semantic features. Exemplarily, the perceptual loss function can be expressed by Formula 2.

[0213]

[0214] Where Lc represents the perceptual loss function, T represents the enhanced image in the image sample pair, represents the predicted image obtained by the second network, φ 1 ,φ 2 ,φ 3 Respectively represent the features extracted from the conv1_2 layer, conv3_2 layer, and conv4_2 layer in VGG-19; λ 1 ,λ 2 ,λ 3 They represent weight coefficients respectively.

[0215] The adversarial loss function is used to enhance the realism of the generated transmission layer results and suppress color cast and reduce artifacts.

[0216]

[0217] Among them, L adv represents the adversarial loss function, T represents the enhanced image in the image sample pair, Represents the predicted image obtained by the second network. Represents the generated image given an input image I is the probability of the true transmission layer image.

[0218] In this embodiment, by using content loss, perceptual loss and adversarial loss to constrain the network, the authenticity of the image restored by the network can be improved, color cast can be suppressed and artifacts can be reduced.

[0219] Step 1405: Train the first network and the second network according to the first loss function to obtain a trained first network and a trained second network.

[0220] The first network and the second network obtained after training in step 1405 can refer to Figure 6 The description in the corresponding embodiments will not be repeated here.

[0221] In a possible embodiment, since the first network is used to predict the tone image, in order to ensure the prediction accuracy of the first network, the terminal can obtain a loss function based on the output and input of the first network, and train the first network and the second network together based on the loss function.

[0222] Specifically, the terminal can obtain the hue image of the first predicted image and the hue image of the enhanced image in the image sample pair, and obtain a second loss function based on the hue image of the first predicted image and the hue image of the enhanced image in the image sample pair, wherein the second loss function is used to indicate the difference between the hue image of the first predicted image and the hue image of the enhanced image in the image sample pair. Finally, the terminal can train the first network and the second network based on the first loss function and the second loss function.

[0223] Since the second loss function is actually obtained based on two tonal images, how to accurately measure the difference between the two tonal images is crucial to network training. Fig.15 , Fig.15 A schematic diagram of a distance measurement of a hue image provided in an embodiment of the present application. Fig.15 As shown, hue is distributed and cyclic in the HSV color space. For any two hue values, the angle between the two hue values ​​is less than 180° under normal circumstances, and is equal to 180° when the angle between the two hue values ​​is at its maximum value. Therefore, the distance between two hue images cannot be directly measured using conventional distance measurement methods (such as L1 distance or L2 distance), but needs to be adjusted accordingly based on the characteristics of the hue.

[0224] Exemplarily, a cyclic tone loss is provided in an embodiment of the present application, which can measure the distance between two tone images. Specifically, the distance measurement method between the tone images is shown in Formula 4 and Formula 5.

[0225]

[0226] M i，j =1when|H a i，j -H b i，j |≤0.5

[0227] M i，j =0when|H a i，j -H b i，j |>0.5 Formula 5

[0228] Among them, H a represents the hue value of any position in the hue image a, represents the hue value of the same position in the hue image b, D hue (H a , H b ) represents the loss function between the tone image a and the tone image b. i，j Represents the weight coefficient.

[0229] Simply put, for the difference of any position in the two toned images, when the angle is less than 0.5, the angle itself is taken, and when the angle is greater than 0.5, the absolute value of the difference between 1 and the angle is taken. Then, the difference corresponding to all positions is summed to obtain the distance between the two toned images, that is, the cyclic toned loss.

[0230] In a possible embodiment, since the hue image can reflect the reflection phenomenon in the image, in order to improve the prediction accuracy of the second network and ensure that the image predicted by the second network can accurately eliminate the reflection phenomenon, a cyclic hue loss can also be introduced to constrain the second network.

[0231] Exemplarily, the terminal may obtain the hue image of the enhanced image in the image sample pair and the third image output by the first network, and then obtain the third loss function based on the hue image of the enhanced image in the image sample pair and the third image, wherein the third loss function is used to indicate the difference between the hue image of the enhanced image in the image sample pair and the third image to obtain the third loss function. The method for obtaining the third loss function is similar to the method for obtaining the second loss function, and specific reference may be made to the above introduction to the second loss function, which will not be repeated here. Finally, the terminal trains the first network and the second network according to the first loss function, the second loss function and the third loss function. That is, the terminal may determine the final total loss function based on the first loss function, the second loss function and the third loss function, and finally train the first network and the second network according to the total loss function.

[0232] Exemplarily, the total loss function is specifically shown in Formula 6 and Formula 7.

[0233] L=α 1 *L hue +α 2 *L 1 +α 2 *L c +α 3 *L adv Formula 6

[0234]

[0235] Among them, L represents the total loss function; L hue represents the total hue loss function, A second loss function corresponding to the hue image of the first predicted image and the hue image of the enhanced image in the image sample pair is represented, represents the third loss function corresponding to the hue image of the enhanced image and the third image in the image sample pair. L1 represents the content loss function. Lc represents the perceptual loss function. adv Denotes the adversarial loss function. α 1 , α 2 , α 3 , β are weight coefficients, and their values ​​can be 50, 0.2, 0.01, and 2 respectively.

[0236] Taking the method of this embodiment for image dereflection as an example, this embodiment is tested on a simulation data set to compare the method with the existing image dereflection algorithm.

[0237] First, in this embodiment, 3500 pairs of reflection images and ground truth (GT) images are obtained by image synthesis. A reflection image refers to an image with a reflection area, and a ground truth image refers to a de-reflected image corresponding to the reflection image. In addition, this embodiment divides multiple groups of test data based on the synthesized images, and the multiple groups of test data are natural-20, real-20, and wild-55. Among them, natural-20 includes 20 pairs of reflection images and GT images, real-20 includes 20 pairs of reflection images and GT images, and wild-55 includes 55 pairs of reflection images and GT images.

[0238] See also Fig.16 , Fig.16 This is a comparative schematic diagram of the network de-reflection effect provided by the embodiment of the present application. Fig.16 As shown, this embodiment uses the same three sets of test data to carry out tests on a traditional network and the network provided by the present application, and obtains the peak signal-to-noise ratio (PPNR) and structural similarity (SSIM) corresponding to the de-reflected images obtained by the traditional network and the network provided by the present application, respectively. Among them, PPNR is an engineering term that represents the ratio of the maximum possible power of a signal to the destructive noise power that affects its representation accuracy, and can be used to measure the quality of an image. SSIM is an indicator used to measure the similarity between two images, and is used to evaluate the quality of the output image processed by the algorithm. Generally speaking, the larger the PPNR corresponding to the image, the higher the quality of the image; the larger the SSIM of the two images, the higher the similarity of the two images. By Fig.16 It can be seen that compared with the traditional network, the image predicted by the network provided in the present application has a larger PPNR and SSIM, which proves that the hue feature information proposed in the embodiment of the present application can be well used to achieve image dereflection.

[0239] See also Fig.17 , Fig.17 A schematic diagram comparing the de-reflection effects of introducing different networks provided in the embodiments of the present application. Fig.17As shown, when the network provided in the embodiment of the present application only includes the above-mentioned second network, the prediction effect of the network is poor; when the network provided in the embodiment of the present application includes the above-mentioned first network and the second network, the prediction effect of the network is improved; when the network provided in the embodiment of the present application includes the above-mentioned first network and the second network, and the cyclic hue loss is used to constrain the network, the prediction effect of the network is the best. In other words, the first network capable of extracting hue feature information and the cyclic hue loss used to constrain the network proposed in the embodiment of the present application have positive benefits for the image dereflection task and can effectively improve the image dereflection effect.

[0240] See also Fig.18 , Fig.18 A schematic diagram comparing the de-reflection effects of different image de-reflection methods provided in the embodiments of the present application. Fig.18 It shows the evaluation results of the image dereflection method provided by the embodiment of the present application and the existing image dereflection method on the public data set, wherein the data in the last row shows the objective index corresponding to the image dereflection method provided by the embodiment of the present application. It can be seen that the objective index corresponding to the image dereflection method provided by the embodiment of the present application exceeds the objective index corresponding to the existing image dereflection method, that is, compared with the existing image dereflection method, the image dereflection method provided by the embodiment of the present application can have a better image dereflection effect.

[0241] See also Fig.19 and Fig. 20 , Fig.19 A comparative schematic diagram of an image de-reflection effect provided in an embodiment of the present application; Fig. 20 A schematic diagram for comparing another image de-reflection effect provided by an embodiment of the present application. Fig.19 It can be seen that the image predicted by the image de-reflection method provided in the embodiment of the present application has higher visual quality and is closer to the true image, and removes most of the reflections, especially the dense streak reflections (such as the enlarged area of ​​the second group of images) and regional reflections (such as the overall background of the third group of images). Fig. 20 It can be seen that compared with the existing image dereflection method, the image dereflection method provided in the embodiment of the present application obtains the result with the cleanest reflection removal and the highest visual quality, which further verifies the superiority of the image dereflection method provided in the embodiment of the present application.

[0242] See also Fig.21 , Fig.21 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.21As shown, an image processing device provided by an embodiment of the present application includes: a processing unit 2101. The processing unit 2101 is used to: input a first image into a first network to obtain one or more groups of intermediate features extracted by the first network, wherein the one or more groups of intermediate features are related to the hue component of an enhanced image of the first image, and the hue component is a component indicating color information in the hue, saturation, and brightness HSV color space; output a second image through processing by the second network, wherein the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more groups of intermediate features.

[0243] In a possible implementation, the processing unit 2101 is further used to extract the one or more groups of intermediate features through one or more first convolutional layers in series in the first network.

[0244] In a possible implementation, the first convolutional layer includes multiple layers of serially connected convolutional layers.

[0245] In a possible implementation, the processing unit 2101 is also used to: extract features of the first image through the second network; fuse the features of the first image and the one or more groups of intermediate features through the second network to obtain one or more groups of fused features; and process the one or more groups of fused features through the second network to obtain the second image.

[0246] In a possible implementation, one or more groups of fused features are obtained by fusing the features of the first image and the one or more groups of intermediate features through one or more feature units in series in the second network; wherein each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fused features output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is a corresponding group of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding group of intermediate features.

[0247] In a possible implementation manner, the second convolutional layer includes multiple layers of parallel connected convolutional layers.

[0248] In a possible implementation, the processing unit 2101 is also used to: process the first image to obtain a hue image corresponding to the first image, the hue image including a hue component of an HSV color space; input the hue image corresponding to the first image into the first network so as to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

[0249] In a possible implementation, the processing unit 2101 is further used to: input the first image into the first network to obtain a hue image of the first image; input the hue image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features, wherein the hue image includes a hue component of the HSV color space.

[0250] In a possible implementation manner, the input of the second network also includes semantic features of the first image.

[0251] In a possible implementation manner, the device is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing, and image de-fogging.

[0252] See also Fig. 22 , Fig. 22 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig. 22 As shown, a model training device provided by an embodiment of the present application includes: an acquisition unit 2201 and a processing unit 2202. The acquisition unit 2201 is used to acquire an image sample pair, wherein the image sample pair includes a first image and an enhanced image of the first image; the processing unit 2202 is used to: input the first image into the first network to obtain one or more sets of intermediate features extracted by the first network, wherein the one or more sets of intermediate features are related to the hue component of the enhanced image of the first image, wherein the hue component is a component indicating color information in the hue, saturation, and brightness HSV color space; output a second image through processing by the second network, wherein the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more sets of intermediate features; acquire a first loss function according to the enhanced image and the second image in the image sample pair, wherein the first loss function is used to indicate the difference between the enhanced image and the second image in the image sample pair; train the first network and the second network according to the first loss function to obtain a trained first network and a trained second network.

[0253] In a possible implementation, the acquisition unit 2201 is further used to acquire the hue image of the second image and the hue image of the enhanced image in the image sample pair; the acquisition unit 2201 is further used to acquire a second loss function based on the hue image of the second image and the hue image of the enhanced image in the image sample pair, and the second loss function is used to indicate the difference between the hue image of the second image and the hue image of the enhanced image in the image sample pair; the processing unit 2202 is further used to train the first network and the second network based on the first loss function and the second loss function.

[0254] In a possible implementation, the acquisition unit 2201 is also used to acquire a hue image of the enhanced image in the image sample pair and a third image output by the first network; the acquisition unit 2201 is also used to acquire a third loss function based on the hue image of the enhanced image in the image sample pair and the third image, and the third loss function is used to indicate the difference between the hue image of the enhanced image in the image sample pair and the third loss function obtained by the third image; the processing unit 2202 is also used to train the first network and the second network based on the first loss function, the second loss function and the third loss function.

[0255] In a possible implementation, the processing unit 2202 is further configured to extract the one or more groups of intermediate features through one or more first convolutional layers in series in the first network.

[0256] In a possible implementation, the first convolutional layer includes multiple layers of serially connected convolutional layers.

[0257] In a possible implementation, the processing unit 2202 is further used to extract features of the first image through the second network; fuse the features of the first image and the one or more groups of intermediate features through the second network to obtain one or more groups of fused features; and process the one or more groups of fused features through the second network to obtain the second image.

[0258] In a possible implementation, the processing unit 2202 is also used to fuse the features of the first image and the one or more groups of intermediate features through one or more feature units in series in the second network to obtain one or more groups of fused features; wherein each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fused features output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is a corresponding group of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding group of intermediate features.

[0259] In a possible implementation manner, the second convolutional layer includes multiple layers of parallel connected convolutional layers.

[0260] In a possible implementation, the processing unit 2202 is also used to process the first image to obtain a hue image corresponding to the first image, wherein the hue image includes a hue component in an HSV color space; and input the hue image corresponding to the first image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

[0261] In a possible implementation, the processing unit 2202 is further used to input the first image into the first network to obtain a hue image of the first image; input the hue image into the first network to be processed by the first network together with the first image to obtain the one or more groups of intermediate features, and the hue image includes a hue component of the HSV color space.

[0262] In a possible implementation manner, the input of the second network also includes semantic features of the first image.

[0263] In a possible implementation manner, the device is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing, and image de-fogging.

[0264] Next, an execution device provided by an embodiment of the present application is introduced. Fig.23 , Fig.23 This is a schematic diagram of a structure of an execution device provided in an embodiment of the present application. The execution device 2300 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Among them, the execution device 2300 can be deployed with Fig.23 The data processing device described in the corresponding embodiment is used to implement Fig.23 The data processing function in the corresponding embodiment. Specifically, the execution device 2300 includes: a receiver 2301, a transmitter 2302, a processor 2303 and a memory 2304 (wherein the number of the processor 2303 in the execution device 2300 can be one or more, Fig.23 In the example of FIG. 2301 , a processor 2303 may include an application processor 23031 and a communication processor 23032. In some embodiments of the present application, the receiver 2301, the transmitter 2302, the processor 2303 and the memory 2304 may be connected via a bus or other means.

[0265] The memory 2304 may include a read-only memory and a random access memory, and provides instructions and data to the processor 2303. A portion of the memory 2304 may also include a non-volatile random access memory (NVRAM). The memory 2304 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0266] The processor 2303 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0267] The method disclosed in the above embodiment of the present application can be applied to the processor 2303, or implemented by the processor 2303. The processor 2303 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 2303. The above processor 2303 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 2303 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 2304, and the processor 2303 reads the information in the memory 2304 and completes the steps of the above method in combination with its hardware.

[0268] The receiver 2301 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 2302 can be used to output digital or character information through the first interface; the transmitter 2302 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 2302 can also include a display device such as a display screen.

[0269] In one embodiment of the present application, the processor 2303 is configured to execute Figure 6 The image enhancement method is executed by the execution device in the corresponding embodiment.

[0270] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0271] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0272] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the image enhancement method described in the above embodiment, or so that the chip in the training device executes the image enhancement method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0273] For details, please refer to Fig.24 , Fig.24A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2400, NPU 2400 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2403, which is controlled by the controller 2404 to extract matrix data from the memory and perform multiplication operations.

[0274] In some implementations, the operation circuit 2403 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2403 is a two-dimensional systolic array. The operation circuit 2403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2403 is a general-purpose matrix processor.

[0275] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2401 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 2408.

[0276] The unified memory 2406 is used to store input data and output data. The weight data is directly transferred to the weight memory 2402 through the storage unit access controller (Direct Memory Access Controller, DMAC) 2405. The input data is also transferred to the unified memory 2406 through the DMAC.

[0277] BIU stands for Bus Interface Unit, i.e., bus interface unit 2413 , which is used for interaction between AXI bus, DMAC and instruction fetch buffer (IFB) 2409 .

[0278] The bus interface unit 2413 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2409 to obtain instructions from the external memory, and is also used for the storage unit access controller 2405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0279] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2406 or to transfer weight data to the weight memory 2402 or to transfer input data to the input memory 2401.

[0280] The vector calculation unit 2407 includes multiple operation processing units, and further processes the output of the operation circuit 2403 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0281] In some implementations, the vector calculation unit 2407 can store the processed output vector to the unified memory 2406. For example, the vector calculation unit 2407 can apply a linear function; or a nonlinear function to the output of the operation circuit 2403, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2403, for example, for use in a subsequent layer in a neural network.

[0282] An instruction fetch buffer 2409 connected to the controller 2404 is used to store instructions used by the controller 2404;

[0283] Unified memory 2406, input memory 2401, weight memory 2402 and instruction fetch memory 2409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0284] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0285] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0286] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0287] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0288] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. An image enhancement method, characterized in that: include: Inputting a first image into a first network, obtaining one or more sets of intermediate features extracted by the first network, wherein the one or more sets of intermediate features are related to a hue component of an enhanced image of the first image, wherein the hue component is a component indicating color information in a hue, saturation, and brightness HSV color space, wherein the first network includes one or more first convolutional layers in series, wherein the first convolutional layer includes a dilated convolutional layer; Through processing by the second network, a second image is output, where the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more groups of intermediate features.

2. The method according to claim 1, characterized in that The step of inputting the first image into the first network to obtain one or more sets of intermediate features extracted by the first network includes: The one or more groups of intermediate features are extracted through one or more first convolutional layers in series in the first network.

3. The method according to claim 2, characterized in that The first convolutional layer includes multiple layers of serially connected convolutional layers.

4. The method according to claim 1, characterized in that: The step of outputting a second image through processing by the second network includes: Extracting features of the first image through the second network; fusing the features of the first image and the one or more sets of intermediate features through the second network to obtain one or more sets of fused features; The one or more sets of fused features are processed by the second network to obtain the second image.

5. The method according to claim 4, characterized in that The fusing the features of the first image and the one or more sets of intermediate features through the second network to obtain fused features includes: fusing the features of the first image and the one or more sets of intermediate features through one or more feature units in series in the second network to obtain one or more sets of fused features; Among them, each of the one or more feature units includes a second convolutional layer and a feature fusion module, the input of the second convolutional layer in each feature unit is the fusion feature output by the feature fusion module in the previous feature unit, the input of the feature fusion module in each feature unit is a corresponding set of intermediate features and the output of the second convolutional layer in the same feature unit, and the feature fusion module in the one or more feature units has a one-to-one corresponding set of intermediate features.

6. The method according to claim 5, characterized in that The second convolutional layer includes multiple layers of convolutional layers connected in parallel.

7. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Processing the first image to obtain a hue image corresponding to the first image, wherein the hue image includes a hue component in an HSV color space; The hue image corresponding to the first image is input into the first network so as to be processed by the first network together with the first image to obtain the one or more groups of intermediate features.

8. The method according to any one of claims 1 to 6, characterized in that: The method further comprises: Inputting the first image into a first network to obtain a hue image of the first image; The hue image is input into the first network to be processed by the first network together with the first image to obtain the one or more sets of intermediate features, wherein the hue image includes a hue component of an HSV color space.

9. The method according to any one of claims 1 to 6, characterized in that: The input of the second network also includes semantic features of the first image.

10. The method according to any one of claims 1 to 6, characterized in that: The method is used to implement at least one of the following image enhancement tasks: image de-reflection, image de-shadowing and image de-fogging.

11. A network model training method, characterized in that: include: Acquire an image sample pair, wherein the image sample pair includes a first image and an enhanced image of the first image; Inputting a first image into a first network, obtaining one or more sets of intermediate features extracted by the first network, wherein the one or more sets of intermediate features are related to a hue component of an enhanced image of the first image, wherein the hue component is a component indicating color information in a hue, saturation, and brightness HSV color space, wherein the first network includes one or more first convolutional layers in series, wherein the first convolutional layer includes a dilated convolutional layer; Outputting a second image through processing by a second network, where the second image is an enhanced image of the first image, and the input of the second network includes the first image and the one or more sets of intermediate features; Acquire a first loss function according to the enhanced image and the second image in the image sample pair, wherein the first loss function is used to indicate a difference between the enhanced image and the second image in the image sample pair; The first network and the second network are trained according to the first loss function to obtain a trained first network and a trained second network.

12. The method according to claim 11, characterized in that The method further comprises: acquiring a tone image of the second image and a tone image of the enhanced image in the image sample pair; Acquire a second loss function according to the hue image of the second image and the hue image of the enhanced image in the image sample pair, wherein the second loss function is used to indicate a difference between the hue image of the second image and the hue image of the enhanced image in the image sample pair; The training of the first network and the second network according to the first loss function includes: The first network and the second network are trained according to the first loss function and the second loss function.

13. The method according to claim 12, characterized in that The method further comprises: Obtaining a hue image of the enhanced image in the image sample pair and a third image output by the first network; Acquire a third loss function according to the hue image of the enhanced image in the image sample pair and the third image, wherein the third loss function is used to indicate a difference between the hue image of the enhanced image in the image sample pair and the third image; The training of the first network and the second network according to the first loss function and the second loss function includes: The first network and the second network are trained according to the first loss function, the second loss function and the third loss function.

14. An image processing device, characterized in that: The image processing apparatus comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the image processing apparatus executes the method according to any one of claims 1 to 13.

15. A computer storage medium, characterized in that The computer storage medium stores instructions, which, when executed by a computer, cause the computer to implement the method of any one of claims 1 to 13.

16. A computer program product, characterized in that The computer program product stores instructions, which, when executed by a computer, cause the computer to implement the method according to any one of claims 1 to 13.

Images