A method of image processing and related apparatus

Abstract

The application discloses a method and related device for image processing, obtaining a first feature map and a second feature map, the first feature map is used for describing high-frequency detail features of a target image, and the second feature map is used for describing low-frequency structural features of the target image, and the first feature map and the second feature map have the same size. The first feature map is subjected to first convolution processing to obtain a first enhancement result, and the second feature map is subjected to second convolution processing to obtain a second enhancement result. The first enhancement result and the second enhancement result are fused to obtain a first fusion result, and the first fusion result is used for generating an enhanced image of the target image. In the application, the high-frequency detail features and the low-frequency structural features of the target image are subjected to convolution processing respectively, wherein the convolution processing of the high-frequency detail features can optimize the detail features of the image, the convolution processing of the low-frequency structural features can optimize the tone mapping of the image, and through the fusion of the two different types of convolution results, the quality of image enhancement can be improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method and related apparatus. Background Technology

[0002] In recent years, with the rapid development of deep learning technology, high dynamic range (HDR) enhancement technology based on neural network algorithms can achieve better enhancement results.

[0003] HDR enhancement technology based on neural network algorithms can compress the size of the image to be processed and extract its details, resulting in a compressed first image and a detail feature map of the original image. High dynamic range (HDR) enhancement is then applied to the first image to obtain an enhanced second image. Based on the detail feature map, the second image is enlarged and its details are restored, resulting in the HDR-enhanced image.

[0004] Comparative tests revealed that existing HDR enhancement algorithms are relatively simple, and the resulting image quality does not adequately meet requirements. Therefore, improving the quality of HDR images is a pressing technical challenge. Summary of the Invention

[0005] This application provides an image processing method and related apparatus for improving the quality of image enhancement.

[0006] In a first aspect, embodiments of this application provide an image processing method. For a target image to be enhanced, it is necessary to obtain a first feature map and a second feature map corresponding to the target image. The first feature map and the second feature map have the same size. The first feature map is used to describe the high-frequency detail features of the target image, and the second feature map is used to describe the low-frequency structural features of the target image.

[0007] In this embodiment, the target image is used as the original image to be enhanced by HDR. Specifically, the target image can be a static SDR image or an image frame from an SDR video stream. That is, the image processing method in this application can be applied to HDR enhancement of static images or HDR enhancement of videos; no specific limitation is made here.

[0008] Since the first feature map describes the high-frequency detail features of the target image, a first convolution process is performed on the first feature map to obtain a first enhancement result, thereby optimizing the image's tone mapping. Since the second feature map describes the low-frequency structural features of the target image, a second convolution process is performed on the second feature map to obtain a second enhancement result, thereby optimizing the image's detail features. Because the processing mechanisms for tone mapping and detail features are different, in this embodiment, the convolutional filter kernels for the two convolution types (first convolution and second convolution) can be different. Therefore, refining the specialized convolutional filter kernel processing based on different task assignments can improve the final enhancement effect.

[0009] The first enhancement result and the second enhancement result are fused together, and the resulting first fusion result can be used to generate an enhanced image of the target image. In this application, high-frequency detail features and low-frequency structural features of the target image are convolved separately. The convolution processing of high-frequency detail features can optimize the detail features of the image, while the convolution processing of low-frequency structural features can optimize the tone mapping of the image. By fusing the two different types of convolution results, the quality of image enhancement can be improved, optimizing both the detail features and the tone mapping of the image.

[0010] Based on the first aspect, in one optional implementation, the first fusion result can continue to be used as input for the next higher level. After upsampling and scaling, a third feature map is obtained, which describes the low-frequency structural features of the target image. Then, the high-frequency structural features of the next higher level, i.e., the fourth feature map, are obtained, where the third and fourth feature maps have the same size. A third convolution process is performed on the third feature map to obtain a third enhancement result, and a fourth convolution process is performed on the fourth feature map to obtain a fourth enhancement result. The third and fourth enhancement results are fused to obtain a second fusion result, which can also be used to generate an enhanced image of the target image.

[0011] In this embodiment, instead of predicting the HDR image of the target image from the outset, the process begins with a small-sized Laplacian pyramid image and proceeds progressively through multiple levels. This ensures that detailed features at each different scale are fully considered. Although the network implements multi-level step-by-step operations to improve performance, the computational complexity is significantly reduced because these scales are lower resolution than the original image. Therefore, the network computation is performed at low resolution, thus not introducing excessive network computational overhead. On the other hand, since no network parameters need to be learned during the Laplacian pyramid construction process, unlike other feature extraction methods that require additional convolutional filters, the computational cost of construction is negligible.

[0012] Based on the first aspect, in an optional implementation, to improve the effect of convolution processing on the second feature map, this embodiment of the application may also employ a target network (feature extraction network) to extract features from the target image to obtain conditional feature maps. If the target image is converted into a Laplacian pyramid with multiple levels, then correspondingly, multiple conditional feature maps with the same number of levels need to be extracted for the target image, wherein the size of the conditional feature map at each level is the same as the size of the high-frequency detail features and low-frequency structural features of the level it belongs to. Convolution processing is performed on the conditional feature map at each level and the low-frequency structural features of the level it belongs to to obtain an enhancement result for the low-frequency structural features.

[0013] Therefore, in this embodiment, a conditional feature map of the same size as the second feature map (low-frequency structural features) can be used to perform convolution processing on the second feature map to obtain the second enhancement result. In this embodiment, based on the conditional feature map as a guide, spatial variability modulation is provided, thereby enabling dynamic adjustment according to local brightness and exposure conditions to optimize overexposed and underexposed areas that may appear in the target image.

[0014] Based on the first aspect, in one optional implementation, the conditional feature map undergoes an affine transformation through a spatial feature transform (SFT) layer to generate γ coefficients and β coefficients. The second feature map is then convolved based on these γ coefficients and β coefficients to obtain the second enhancement result.

[0015] Based on the first aspect, in one optional implementation, the aforementioned dual-branch convolutional fusion operation can be performed on the Laplacian pyramid image at each level until the top layer of the Laplacian pyramid. As the top layer of the Laplacian pyramid, after the fusion result of the previous level is upsampled and enlarged, the size of the low-frequency structural feature is now the same as the size of the target image, while the high-frequency detail feature is the Laplacian pyramid image at the same level. At this point, convolutional processing of the low-frequency structural feature and the high-frequency detail feature is performed on the current top layer separately, and then the two enhancement results are fused to obtain the final enhanced image of the target image. In other words, if the third structural feature obtained after upsampling the first fusion result has the same size as the target image, then the aforementioned second fusion result can be used as the enhanced image of the target image.

[0016] In this embodiment, the image processing flow starts from the smallest layer of the Laplacian pyramid. The fusion result obtained from each layer is input to the next layer, and after upsampling, it continues to serve as the low-frequency structural features of the next layer, while the high-frequency detail features are the Laplacian pyramid image of the same layer. Each layer repeatedly performs convolution and fusion operations until the processing of the last layer is completed. At this point, the size of the fusion result is the same as the size of the target image, and this fusion result can be used as the enhanced image of the target image. It should be noted that the Gaussian convolution kernel and the number of decomposition layers in the Laplacian pyramid can be preset and adjusted according to the actual application scenario. Furthermore, although the network structure is the same at each layer, the dual-path residual modules of each layer have their own convolution filter parameters. This is because each layer of the Laplacian pyramid represents different components of the original input signal (i.e., the target image) in the spectrum, and the network learns an adaptive mapping for the presence of different components through training.

[0017] Secondly, embodiments of this application provide an image processing apparatus, including:

[0018] The acquisition unit is used to acquire a first feature map and a second feature map. The first feature map is used to describe the high-frequency detail features of the target image, and the second feature map is used to describe the low-frequency structural features of the target image. The first feature map and the second feature map have the same size.

[0019] The processing unit is used to perform a first convolution process on the first feature map to obtain a first enhancement result;

[0020] The processing unit is also used to perform a second convolution process on the second feature map to obtain a second enhancement result;

[0021] The processing unit is further configured to fuse the first enhancement result and the second enhancement result to obtain a first fusion result, which is used to generate an enhanced image of the target image.

[0022] The information interaction and execution process of the embodiments shown in this aspect are based on the same concept as the embodiments shown in the first aspect. Therefore, for a detailed explanation of the beneficial effects shown in this aspect, please refer to the first aspect above, and will not be repeated here.

[0023] Based on the second aspect, in an optional implementation, the processing unit is further configured to upsample the first fusion result to obtain a third feature map, the third feature map being used to describe the low-frequency structural features of the target image.

[0024] The acquisition unit is also used to acquire a fourth feature map, which is used to describe the high-frequency detail features of the target image. The third feature map and the fourth feature map have the same size.

[0025] The processing unit is also used to perform a third convolution on the third feature map to obtain a third enhancement result;

[0026] The processing unit is also used to perform a fourth convolution on the fourth feature map to obtain a fourth enhancement result;

[0027] The processing unit is also used to fuse the third enhancement result and the fourth enhancement result to obtain a second fusion result, which is used to generate an enhanced image of the target image.

[0028] Based on the second aspect, in one optional implementation, the processing unit is specifically used for:

[0029] The second feature map is subjected to a second convolution process based on the conditional feature map to obtain the second enhancement result. The conditional feature map is obtained by extracting features from the target image through the target network, and the conditional feature map and the second feature map have the same size.

[0030] Based on the second aspect, in one optional implementation, the processing unit is specifically used for:

[0031] The conditional feature map is subjected to an affine transformation SFT to obtain the γ coefficients and β coefficients;

[0032] The second feature map is subjected to a second convolution process based on the γ and β coefficients.

[0033] Based on the second aspect, in an optional embodiment, the image processing apparatus further includes:

[0034] The determining unit is used to determine the second fusion result as the enhanced image of the target image when the third feature map has the same size as the target image.

[0035] Thirdly, embodiments of the present invention provide a computer device, including a memory, a communication interface, and a processor coupled to the memory and the communication interface; the memory is used to store instructions, the processor is used to execute the instructions, and the communication interface is used to communicate with other devices under the control of the processor; wherein, when the processor executes the instructions, it performs the image processing method described in any of the preceding aspects.

[0036] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the image processing method described in any of the preceding aspects.

[0037] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions that, when run on a computer, cause the computer to perform the image processing method described in any of the preceding aspects.

[0038] Sixthly, embodiments of this application provide a chip system including a processor for implementing the functions involved in the foregoing aspects, such as transmitting or processing data and / or information involved in the foregoing methods. In one possible design, the chip system further includes a memory for storing program instructions and data necessary for a server or communication device. This chip system may be composed of chips or may include chips and other discrete devices.

[0039] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0040] This application discloses an image processing method and related apparatus. The method involves acquiring a first feature map and a second feature map. The first feature map describes high-frequency detail features of a target image, and the second feature map describes low-frequency structural features of the target image. The first and second feature maps have the same size. A first convolution process is performed on the first feature map to obtain a first enhancement result, and a second convolution process is performed on the second feature map to obtain a second enhancement result. The first and second enhancement results are then fused to obtain a first fusion result, which is used to generate an enhanced image of the target image. In this application, convolution processes are performed on both the high-frequency detail features and the low-frequency structural features of the target image. The convolution process on the high-frequency detail features optimizes the image's detail features, while the convolution process on the low-frequency structural features optimizes the image's tone mapping. By fusing these two different types of convolution results, the quality of image enhancement can be improved. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0042] Figure 1 A structural diagram illustrating the main framework of artificial intelligence;

[0043] Figure 2 A schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0044] Figure 3 Another schematic diagram of a convolutional neural network provided in an embodiment of this application;

[0045] Figure 4 A schematic diagram illustrating the framework of different HDR enhancement technologies;

[0046] Figure 5 This is a flowchart illustrating the image processing method in an embodiment of this application;

[0047] Figure 6 This is a schematic diagram illustrating the process of generating multiple levels of high-frequency detail features and low-frequency structural features in the embodiments of this application.

[0048] Figure 7 This is a schematic diagram illustrating the second convolution process performed on the second feature map based on the conditional feature map in an embodiment of this application;

[0049] Figure 8 This is a schematic diagram of image processing at a single level in an embodiment of this application;

[0050] Figure 9 This is a schematic diagram of the image processing flow based on the Laplacian pyramid in an embodiment of this application;

[0051] Figure 10 This is a schematic diagram illustrating multiple levels of image processing in the embodiments of this application;

[0052] Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application;

[0053] Figure 12 This is a schematic diagram of a computing device for HDR enhancement of video images in an embodiment of this application;

[0054] Figure 13 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0055] This application provides an image processing method and related apparatus for improving the quality of image enhancement.

[0056] The embodiments of the present invention will now be described with reference to the accompanying drawings. The terminology used in the embodiments section is for illustrative purposes only and is not intended to limit the scope of the invention. As will be apparent to those skilled in the art, with technological advancements and the emergence of new scenarios, the technical solutions provided in this application are equally applicable to similar technical problems.

[0057] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0058] First, the overall workflow of the artificial intelligence system is described; please refer to [link / reference]. Figure 1 , Figure 1 The diagram illustrates a structural framework for artificial intelligence (AI). The framework is further elaborated below along two dimensions: the "Intelligent Information Chain" (horizontal axis) and the "IT Value Chain" (vertical axis). The "Intelligent Information Chain" reflects a series of processes from data acquisition to processing. For example, it could be the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. In this process, data undergoes a condensation process of "data—information—knowledge—wisdom." The "IT Value Chain" reflects the value that AI brings to the information technology industry, from the underlying infrastructure of human intelligence and information (provided and processed through technological means) to the industrial ecosystem of the system.

[0059] (1) Infrastructure.

[0060] Infrastructure provides computing power to support artificial intelligence systems, enabling communication with the external world and providing support through a basic platform. This communication occurs through sensors; computing power is provided by intelligent chips (hardware acceleration chips such as CPUs, NPUs, GPUs, ASICs, and FPGAs); and the basic platform includes distributed computing frameworks and related platform guarantees and support, which may include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to acquire data, and this data is provided to intelligent chips in the distributed computing system provided by the basic platform for computation.

[0061] (2) Data.

[0062] The data at the next layer of infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data from traditional devices, including business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0063] (3) Data processing.

[0064] Data processing typically includes methods such as data training, machine learning, deep learning, search, reasoning, and decision-making.

[0065] Among them, machine learning and deep learning can perform intelligent information modeling, extraction, preprocessing, and training on data, including symbolization and formalization.

[0066] Reasoning refers to the process in which, in a computer or intelligent system, the machine thinks and solves problems by simulating human intelligent reasoning, based on reasoning control strategies and using formalized information. Typical functions include search and matching.

[0067] Decision-making refers to the process of making decisions based on intelligent information after reasoning, and it typically provides functions such as classification, sorting, and prediction.

[0068] (4) General ability.

[0069] After the data processing mentioned above, the results of the data processing can be used to form some general capabilities, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0070] (5) Smart products and industry applications.

[0071] Intelligent products and industry applications refer to products and applications of artificial intelligence systems in various fields. They are the encapsulation of overall artificial intelligence solutions, productizing intelligent information decision-making and realizing practical applications. Their application areas mainly include: intelligent terminals, intelligent transportation, intelligent healthcare, autonomous driving, smart cities, etc.

[0072] The method provided in this application is described below from the perspectives of model training and model application:

[0073] The model training method provided in this application can be applied to data processing methods such as data training, machine learning, and deep learning. It performs symbolic and formal intelligent information modeling, extraction, preprocessing, and training on training data to finally obtain a trained neural network model (such as the target neural network model in this application embodiment). The target neural network model can be used for model inference. Specifically, input data can be input into the target neural network model to obtain output data.

[0074] Since the embodiments of this application involve a large number of neural network applications, for ease of understanding, the relevant terms and concepts such as neural networks involved in the embodiments of this application will be introduced below.

[0075] (1) Neural network.

[0076] A neural network can be composed of neural units, which can be defined as a computational unit that takes xs (i.e., input data) and an intercept of 1 as input. The output of this computational unit can be:

[0077] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into an output signal. The output signal of this activation function can be used as the input of the next convolutional layer, and the activation function can be the sigmoid function. A neural network is a network formed by connecting multiple of the above-mentioned individual neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, which can be a region composed of several neural units.

[0078] (2) A Convolutional Neural Network (CNN) is a deep neural network with a convolutional structure. A CNN contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer refers to the layer of neurons in a CNN that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can be connected to only some of the neurons in its neighboring layers. A convolutional layer typically contains several feature planes, each of which can be composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernels. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of the image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, for all locations on the image, we can use the same learned image information. In the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolution operation.

[0079] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0080] Specifically, such as Figure 2 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0081] The structure consisting of the convolutional layer / pooling layer 120 and the neural network layer 130 can be the first convolutional layer and the second convolutional layer described in this application. The input layer 110 is connected to the convolutional layer / pooling layer 120, and the convolutional layer / pooling layer 120 is connected to the neural network layer 130. The output of the neural network layer 130 can be input to the activation layer, and the activation layer can perform non-linear processing on the output of the neural network layer 130.

[0082] Convolutional / pooling layers: 120. Convolutional layers: such as... Figure 2 The convolutional / pooling layer 120 shown may include layers 121-126 as in Examples 121-126. In one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer. In another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer to continue the convolution operation.

[0083] Taking convolutional layer 121 as an example, it can include multiple convolution operators, also known as kernels. In image processing, a convolution operator acts as a filter, extracting specific information from the input image matrix. Essentially, a convolution operator can be a weight matrix, which is usually predefined. During the convolution operation, the weight matrix processes the input image pixel by pixel (or two pixels by two pixels, depending on the stride) along the horizontal direction, thus extracting specific features. The size of the weight matrix should be related to the image size. It's important to note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image; during convolution, the weight matrix extends to the entire depth of the input image. Therefore, convolution with a single weight matrix produces a single-depth convolutional output. However, in most cases, multiple weight matrices of the same dimension are applied instead of a single weight matrix. The outputs of each weight matrix are stacked to form the depth dimension of the convolutional image. Different weight matrices can be used to extract different features from an image. For example, one weight matrix can be used to extract image edge information, another weight matrix can be used to extract specific colors from the image, and yet another weight matrix can be used to blur unwanted noise in the image. These multiple weight matrices have the same dimension, and the feature maps extracted by these multiple weight matrices with the same dimension also have the same dimension. The extracted feature maps with the same dimension are then merged to form the output of the convolution operation.

[0084] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0085] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0086] Pooling layers: Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 2 In the example of 120, each layer 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers.

[0087] Neural Network Layer 130: After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required output information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layer 130 to generate one or a set of required class numbers of output. Therefore, neural network layer 130 may include multiple hidden layers (such as...). Figure 2 As shown in 131, 132 to 13n) and output layer 140, the parameters contained in these multi-layer hidden layers can be pre-trained based on relevant training data for specific task types, such as image recognition, image classification, image super-resolution reconstruction, etc.

[0088] After the multiple hidden layers in neural network layer 130, the final layer of the entire convolutional neural network 100 is the output layer 140. This output layer 140 has a loss function similar to classification cross-entropy, specifically used to calculate the prediction error. Once the entire convolutional neural network 100 has undergone forward propagation (e.g., ...), the loss function is applied. Figure 2 The propagation from 110 to 140 is completed (forward propagation), and the reverse propagation (such as...) Figure 2 The propagation from 140 to 110 (backpropagation) will begin to update the weight values ​​and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the output of the convolutional neural network 100 through the output layer and the ideal result.

[0089] It should be noted that, as Figure 2 The convolutional neural network 100 shown is merely an example of a convolutional neural network. In specific applications, convolutional neural networks can also exist in the form of other network models, such as... Figure 3 The multiple convolutional / pooling layers shown are run in parallel, and the extracted features are all input into the full neural network layer 130 for processing.

[0090] (3) Deep neural networks.

[0091] Deep Neural Networks (DNNs), also known as multilayer neural networks, can be understood as neural networks with many hidden layers, though there's no specific metric for "many." DNNs can be categorized into three layers based on their position: input layers, hidden layers, and output layers. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. All layers are fully connected, meaning that any neuron in the i-th layer is connected to any neuron in the (i+1)-th layer. Although DNNs appear complex, the operation of each layer is actually quite simple, resembling a linear relationship as follows: in, It is the input vector. It is the output vector. α is the offset vector, W is the weight matrix (also called coefficients), and α() is the activation function. Each layer is simply an adjustment of the input vector. The output vector is obtained through such a simple operation. Because DNNs have many layers, the coefficients W and the offset vector... The number of these parameters is therefore quite large. The definitions of these parameters in a DNN are as follows: Taking the coefficient W as an example: Assuming a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as... The superscript 3 represents the layer number where coefficient W resides, while the subscript corresponds to the output third layer index 2 and the input second layer index 4. In summary, the coefficients from the k-th neuron in layer L-1 to the j-th neuron in layer L are defined as follows: It's important to note that the input layer does not have a W parameter. In deep neural networks, more hidden layers allow the network to better represent complex real-world situations. Theoretically, the more parameters a model has, the higher its complexity and "capacity," meaning it can perform more complex learning tasks. Training a deep neural network is essentially the process of learning the weight matrix, with the ultimate goal of obtaining the weight matrix of all layers in the trained deep neural network (a weight matrix formed by the vectors W from many layers).

[0092] (4) Loss function.

[0093] In training a deep neural network, to ensure the output closely approximates the desired predicted value, we compare the network's prediction with the target value. Based on the difference, we update the weight vector of each layer (usually pre-configuring parameters before the initial update). For example, if the prediction is too high, the weight vector is adjusted to predict a lower value. This adjustment continues until the deep neural network predicts the target value or a value very close to it. Therefore, we need to predefine "how to compare the difference between the predicted and target values," which is the loss function or objective function. These are important equations used to measure the difference between the predicted and target values. Taking the loss function as an example, a higher output value (loss) indicates a greater difference, and training the deep neural network becomes a process of minimizing this loss.

[0094] (5) Backpropagation algorithm.

[0095] Convolutional neural networks can employ backpropagation (S21P000217) to correct the parameters in the initial super-resolution model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss; this error loss information is then propagated back to update the parameters in the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining the optimal parameters of the super-resolution model, such as the weight matrix.

[0096] (6) Linear operations.

[0097] Linearity refers to a proportional, linear relationship between quantities. Mathematically, it can be understood as a function whose first derivative is a constant. Linear operations include, but are not limited to, addition, empty operations, identity operations, convolution, batch normalization (BN), and pooling. Linear operations can also be called linear mappings. Linear mappings must satisfy two conditions: homogeneity and additivity. If either condition is not met, it is nonlinear.

[0098] In recent years, with the rapid development of deep learning technology, high dynamic range (HDR) enhancement technology based on neural network algorithms can achieve better enhancement results.

[0099] Please see Figure 4 , Figure 4A schematic diagram illustrating the framework of different HDR enhancement technologies. (e.g.) Figure 4 As shown on the left, traditional linear domain HDR enhancement techniques typically operate in the linear domain of image illumination, predicting missing HDR brightness information by receiving images with standard dynamic range (SDR) brightness. However, in applications such as film and television or games, the input SDR video sequence only contains pixel value information, thus limiting the application of traditional linear domain HDR enhancement techniques. For example... Figure 4 As shown on the right, in the pixel-domain HDR enhancement technology based on neural network algorithms, by inputting SDR images or videos into a neural network model, end-to-end reconstruction can be directly performed from the pixel domain of the SDR video, and the HDR enhancement result can be output.

[0100] Existing HDR enhancement techniques based on neural network algorithms can compress the image to be processed and extract its details, resulting in a compressed first image and a detail feature map of the original image. High dynamic range (HDR) enhancement is then applied to the first image to obtain an enhanced second image. Based on the detail feature map, the second image is enlarged and its details are restored, resulting in the HDR-enhanced image.

[0101] Comparative tests revealed that existing HDR enhancement algorithms are relatively simple, and the resulting image quality does not adequately meet requirements. Therefore, improving the quality of HDR images is a pressing technical challenge.

[0102] In view of this, embodiments of this application provide an image processing method for improving the quality of image enhancement. Please refer to... Figure 5 , Figure 5 This is a flowchart illustrating the image processing method in an embodiment of this application, as shown below. Figure 5 As shown, the image processing method in this embodiment includes:

[0103] 201. Obtain the first feature map and the second feature map.

[0104] In this embodiment, the target image is used as the original image to be enhanced by HDR. Specifically, the target image can be a static SDR image or an image frame from an SDR video stream. That is, the image processing method in this application can be applied to HDR enhancement of static images or HDR enhancement of videos; no specific limitation is made here.

[0105] For the target image to be enhanced, it is necessary to obtain the first feature map and the second feature map corresponding to the target image. The first feature map and the second feature map have the same size. The first feature map is used to describe the high-frequency detail features of the target image, and the second feature map is used to describe the low-frequency structural features of the target image.

[0106] In practical applications, to further improve the quality of image enhancement, the target image can be transformed using a Laplacian pyramid to generate high-frequency detail features and low-frequency structural features at multiple sizes. The lowest level of the Laplacian pyramid, after Gaussian downsampling and then upsampling, represents the low-frequency structural features of that lowest level. The image processing method provided in this application can be applied to the Laplacian pyramid at any level of the target image; that is, the first feature map and the second feature map in this application can be the high-frequency detail features and low-frequency structural features of the Laplacian pyramid at any level of the target image.

[0107] For easier understanding, please refer to Figure 6 , Figure 6 This is a schematic diagram illustrating the process of generating multiple levels of high-frequency detail features and low-frequency structural features in embodiments of this application. For example... Figure 6 As shown, the input image at the current level is first blurred using a Gaussian filter kernel, followed by downsampling and interpolation upsampling (scale x2 variation). At this point, the image is double-blurred. Subtracting the double-blurred image from the input image yields the high-frequency detail features. The downsampled image of the current level's input image serves as the input image for the next layer of Laplacian decomposition. The same residual separation operation is performed at each layer until the preset number of pyramid layers is reached. By recording the high-frequency detail features and low-frequency structural features at each layer, the Laplacian pyramid of the target image can be obtained.

[0108] It should be understood that the embodiments of this application do not limit the specific methods for obtaining high-frequency detail features and low-frequency structural features. In practical applications, in addition to using the Laplacian pyramid transformation method described above, other upsampling and downsampling methods can also be used to obtain high-frequency detail features and low-frequency structural features of the target image. The specific methods are not limited here.

[0109] 202. Perform a first convolution process on the first feature map to obtain a first enhancement result.

[0110] Since the first feature map describes the high-frequency detail features of the target image, the first convolution process is performed on the first feature map to obtain the first enhancement result, thereby optimizing the tone mapping of the image.

[0111] 203. Perform a second convolution on the second feature map to obtain the second enhancement result.

[0112] Since the second feature map describes the low-frequency structural features of the target image, a second convolution process is performed on the second feature map to obtain a second enhancement result, thereby optimizing the image's detail features. Because the processing mechanisms for tone mapping and detail features are different, in this embodiment, the convolutional filter kernels for the two convolution types (first convolution and second convolution) can be different. Therefore, refining the specialized convolutional filter kernel processing based on different task assignments can improve the final enhancement effect.

[0113] It should be understood that this application does not limit the execution order of steps 202 and 203. That is, step 202 can be executed first and then step 203; step 203 can be executed first and then step 202; or steps 202 and 203 can be executed simultaneously. No specific limitation is made here.

[0114] Furthermore, to improve the effect of convolution processing on the second feature map, in this embodiment, a target network (feature extraction network) can also be used to extract features from the target image to obtain conditional feature maps. If the target image is converted into a Laplacian pyramid with multiple levels, then correspondingly, multiple conditional feature maps with the same number of levels need to be extracted for the target image. The size of each level's conditional feature map is the same as the size of its high-frequency detail features and low-frequency structural features. Convolution processing is performed on each level's conditional feature map and its low-frequency structural features to obtain an enhancement result targeting those low-frequency structural features.

[0115] Therefore, in this embodiment, a conditional feature map of the same size as the second feature map (low-frequency structural features) can be combined to perform convolution processing on the second feature map to obtain the second enhancement result. For details, please refer to... Figure 7 , Figure 7 This is a schematic diagram illustrating the second convolution process performed on the second feature map based on the conditional feature map in an embodiment of this application. For example... Figure 7 As shown, the conditional feature map undergoes an affine transformation through a spatial feature transform (SFT) layer to generate γ and β coefficients. The second feature map is then convolved based on these γ and β coefficients to obtain the second enhancement result.

[0116] In this embodiment, based on the conditional feature map as a guide, spatial variability modulation is provided, thereby enabling dynamic adjustment according to local brightness and exposure conditions to optimize overexposed and underexposed areas that may appear in the target image.

[0117] It should be understood that the above-described convolution process of the second feature map based on the conditional feature map is also applicable to the convolution process of other levels of the Laplacian pyramid of the target image for low-frequency structural features. That is, the conditional feature map of each level can be convolved with the low-frequency structural features of the level to obtain the enhancement result for the low-frequency structural features. The specifics will not be elaborated here.

[0118] 204. The first enhancement result and the second enhancement result are fused together to obtain the first fused result.

[0119] The first enhancement result and the second enhancement result are fused together. The resulting first fused result can be used to generate an enhanced image of the target image in subsequent processes. For easier understanding, please refer to [link to relevant documentation]. Figure 8 , Figure 8 This is a schematic diagram of image processing at a single level in an embodiment of this application. For example... Figure 8 As shown, convolution processing is performed on high-frequency detail features and low-frequency structural features respectively. Furthermore, after affine transformation of the conditional feature map corresponding to each level, the generated γ and β coefficients are applied to the convolution processing of low-frequency structural features. Finally, the two enhancement results of the high-frequency detail features and low-frequency structural features are fused.

[0120] In this application, high-frequency detail features and low-frequency structural features of the target image are convolved separately. The convolution processing of high-frequency detail features can optimize the detail features of the image, while the convolution processing of low-frequency structural features can optimize the tone mapping of the image. By fusing the results of the two different types of convolution, the quality of image enhancement can be improved, which optimizes both the detail features and the tone mapping of the image.

[0121] Furthermore, in this embodiment, the first fusion result can continue to be used as input for the next higher level. After upsampling and scaling, a third feature map is obtained, which describes the low-frequency structural features of the target image. Then, the high-frequency structural features of the next higher level, namely the fourth feature map, are obtained, where the third and fourth feature maps have the same size. A third convolution process is performed on the third feature map to obtain a third enhancement result, and a fourth convolution process is performed on the fourth feature map to obtain a fourth enhancement result. The third and fourth enhancement results are fused to obtain a second fusion result, which can also be used to generate an enhanced image of the target image.

[0122] In this embodiment, instead of predicting the HDR image of the target image from the outset, the process begins with a small-sized Laplacian pyramid image and proceeds progressively through multiple levels. This ensures that detailed features at each different scale are fully considered. Although the network implements multi-level step-by-step operations to improve performance, the computational complexity is significantly reduced because these scales are lower resolution than the original image. Therefore, the network computation is performed at low resolution, thus not introducing excessive network computational overhead. On the other hand, since no network parameters need to be learned during the Laplacian pyramid construction process, unlike other feature extraction methods that require additional convolutional filters, the computational cost of construction is negligible.

[0123] Please see Figure 9 , Figure 9 This is a schematic diagram of the image processing flow based on the Laplacian pyramid in an embodiment of this application. Figure 9 As shown, the above-described dual-branch convolution and fusion operation can be performed on the Laplacian pyramid image at each level, up to the top level of the Laplacian pyramid. As the top level of the Laplacian pyramid, after the fusion result from the previous level is upsampled and enlarged, the size of the low-frequency structural feature is now the same as the size of the target image, while the high-frequency detail feature is the Laplacian pyramid image at the same level. At this point, convolution processing of the low-frequency structural feature and the high-frequency detail feature is performed on the current top level separately, and then the two enhancement results are fused to obtain the final enhanced image of the target image. In other words, if the third structural feature obtained after upsampling the first fusion result has the same size as the target image, then the above second fusion result can be used as the enhanced image of the target image.

[0124] For easier understanding, please refer to Figure 10 , Figure 10 This is a schematic diagram illustrating multiple levels of image processing in an embodiment of this application. For example... Figure 10As shown, the image processing flow in this embodiment starts from the smallest layer of the Laplacian pyramid. The fusion result obtained from each layer is input to the next layer, and after upsampling, it continues to serve as the low-frequency structural features of the next layer, while the high-frequency detail features are the Laplacian pyramid image of the same layer. Each layer repeatedly performs convolution and fusion operations until the processing of the last layer is completed. At this point, the size of the fusion result is the same as the size of the target image, and the fusion result can be used as the enhanced image of the target image. It should be noted that the Gaussian convolution kernel and the number of decomposition layers in the Laplacian pyramid can be preset and adjusted according to the actual application scenario. Furthermore, although the network structure of each layer is the same, the dual-path residual module of each layer has its own convolution filter parameters. This is because each layer of the Laplacian pyramid represents different components of the original input signal (i.e., the target image) in the spectrum, and the network learns an adaptive mapping for the existence of different components through training.

[0125] In this embodiment, instead of predicting the HDR image of the target image from the outset, the process begins with a small-sized Laplacian pyramid image and proceeds progressively through multiple levels. This ensures that detailed features at each different scale are fully considered. Although the network implements multi-level step-by-step operations to improve performance, the computational complexity is significantly reduced because these scales are lower resolution than the original image. Therefore, the network computation is performed at low resolution, thus not introducing excessive network computational overhead. On the other hand, since no network parameters need to be learned during the Laplacian pyramid construction process, unlike other feature extraction methods that require additional convolutional filters, the computational cost of construction is negligible.

[0126] Comparative tests show that the image processing method in this application significantly improves both image processing quality and processing speed. Details are shown in Table 1 below:

[0127] Image processing methods JSI-GAN Baseline Based on photomask Flow model This application PSNR (Image Quality) / DB 31.81 32.24 32.80 34.46 4K image processing speed 3.93 4.88 6.87 1.85

[0128] Table 1

[0129] exist Figure 5 Based on the corresponding embodiments, in order to better implement the above-described solutions of this application, related equipment for implementing the above solutions is also provided below. For details, please refer to... Figure 11 , Figure 11 This is a schematic diagram of the structure of an image processing apparatus provided in an embodiment of this application. The image processing apparatus includes:

[0130] The acquisition unit 301 is used to acquire a first feature map and a second feature map. The first feature map is used to describe the high-frequency detail features of the target image, and the second feature map is used to describe the low-frequency structural features of the target image. The first feature map and the second feature map have the same size.

[0131] Processing unit 302 is used to perform a first convolution process on the first feature map to obtain a first enhancement result;

[0132] The processing unit 302 is further configured to perform a second convolution process on the second feature map to obtain a second enhancement result;

[0133] The processing unit 302 is further configured to fuse the first enhancement result and the second enhancement result to obtain a first fusion result, which is used to generate an enhanced image of the target image.

[0134] The information interaction and execution process of the embodiments shown in this aspect are based on the same concept as the embodiments shown in the first aspect. Therefore, for a detailed explanation of the beneficial effects shown in this aspect, please refer to the first aspect above, and will not be repeated here.

[0135] In one possible design, the processing unit 302 is also used to upsample the first fusion result to obtain a third feature map, which is used to describe the low-frequency structural features of the target image.

[0136] The acquisition unit 301 is also used to acquire a fourth feature map, which is used to describe the high-frequency detail features of the target image. The third feature map and the fourth feature map have the same size.

[0137] The processing unit 302 is also used to perform a third convolution process on the third feature map to obtain a third enhancement result;

[0138] The processing unit 302 is also used to perform a fourth convolution process on the fourth feature map to obtain a fourth enhancement result;

[0139] The processing unit 302 is further configured to fuse the third enhancement result and the fourth enhancement result to obtain a second fusion result, which is used to generate an enhanced image of the target image.

[0140] In one possible design, the processing unit 302 is specifically used for:

[0141] The second feature map is subjected to a second convolution process based on the conditional feature map to obtain the second enhancement result. The conditional feature map is obtained by extracting features from the target image through the target network, and the conditional feature map and the second feature map have the same size.

[0142] In one possible design, the processing unit 302 is specifically used for:

[0143] The conditional feature map is subjected to an affine transformation SFT to obtain the γ coefficients and β coefficients;

[0144] The second feature map is subjected to a second convolution process based on the γ and β coefficients.

[0145] In one possible design, the image processing device also includes:

[0146] The determining unit 303 is used to determine the second fusion result as the enhanced image of the target image when the third feature map has the same size as the target image.

[0147] It should be noted that the information interaction and execution process between the modules / units in the image processing device are different from those in this application. Figure 5 The corresponding method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0148] In this embodiment, the target image can be a static SDR image or an image frame from an SDR video stream. That is, the image processing method in this application can be applied to HDR enhancement of static images or HDR enhancement of videos; no specific limitation is made here. For example, regarding HDR enhancement of an SDR video stream, please refer to [link to relevant documentation]. Figure 12 , Figure 12 This is a schematic diagram of a computing device for HDR enhancement of video images in an embodiment of this application. Figure 12 As shown, the program code is supported by a deep learning framework (4011) and deployed on a computing device (4001). The input to this application framework is the SDR signal acquired by different hardware or software (rendering engines, decoders, or sensors, etc.) and stored in the data cache (4021). At runtime, the program code of this application runs in GPU memory (4022). The parts shown in the dashed boxes are the modules newly added (optimized) in the HDR video enhancement network model (4012) of this application: pyramid construction module (4013), feature extraction module (4014), and dual-path residual module (4015).

[0149] The data preprocessing module (4010) first performs vectorization and normalization operations on the data to be processed in the cache, and then sends it to the pyramid construction module (4013) and feature extraction module (4014) in the HDR enhancement network. The pyramid construction module (4013) generates the components of the input SDR frame at different scales, and the feature extraction module (4014) extracts the conditional feature maps of the input frame at different scales. After the construction and extraction are completed, starting from the smallest pyramid scale layer, the dual-path residual module (4015) receives the components of that pyramid layer and the corresponding conditional feature map, and generates HDR enhanced frames layer by layer. The final HDR result is then output to the output cache (4023).

[0150] The logical (operation and connection) relationship between the hardware and software is as follows: The data cache (4021) is used to store SDR signals generated / captured by various hardware / software (rendering engine, decoder, sensor) to prepare for the input of the method. The GPU (4022) provides computational acceleration inference support for the deep learning framework (4011).

[0151] This application also provides a computing device; please refer to [link / reference]. Figure 13 , Figure 13 This is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 500 may be equipped with... Figure 11 In the corresponding embodiment of the image processing apparatus, specifically, the computing device 500 is implemented by one or more servers. The computing device 500 can vary significantly due to differences in configuration or performance, and may include one or more central processing units (CPUs) 522 (e.g., one or more processors) and memory 532, and one or more storage media 530 (e.g., one or more mass storage devices) for storing application programs 542 or data 544. The memory 532 and storage media 530 can be temporary or persistent storage. The program stored in the storage media 530 may include one or more modules (not shown in the figure), each module including a series of instruction operations on the computing device. Furthermore, the CPU 522 may be configured to communicate with the storage media 530 and execute the series of instruction operations in the storage media 530 on the computing device 500.

[0152] The computing device 500 may also include one or more power supplies 526, one or more wired or wireless network interfaces 550, one or more input / output interfaces 558, and / or one or more operating systems 541, such as Windows Server. TM Mac OS X TM Unix TMLinux TM FreeBSD TM etc.

[0153] It should be noted that the information interaction and execution process between the various modules / units in the computing device are different from those in this application. Figure 5 The corresponding method embodiments are based on the same concept, and the details can be found in the descriptions of the method embodiments shown above in this application, which will not be repeated here.

[0154] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figure 5 The method described in the illustrated embodiment.

[0155] This application embodiment also provides a computer-readable storage medium storing a program for performing signal processing, which, when run on a computer, causes the computer to perform the aforementioned actions. Figure 5 The method described in the illustrated embodiment.

[0156] The image processing device provided in this application embodiment can specifically be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in the storage unit. Optionally, the storage unit can be a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0157] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0158] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0159] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0160] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).

Claims

1. A method of image processing, characterized by, include: A first feature map and a second feature map are obtained. The first feature map is used to describe the high-frequency detail features of the target image, and the second feature map is used to describe the low-frequency structural features of the target image. The first feature map and the second feature map have the same size. The first feature map is subjected to a first convolution process to obtain a first enhancement result, wherein the convolutional filter kernel of the first convolution process is used to optimize the tone mapping of the image; The second feature map is subjected to a second convolution process to obtain a second enhancement result. The convolutional filter kernel of the second convolution process is used to optimize the detailed features of the image, and the convolutional filter kernel of the first convolution process and the convolutional filter kernel of the second convolution process are different from each other. The first enhancement result and the second enhancement result are fused to obtain a first fusion result, which is used to generate an enhanced image of the target image.

2. The method of claim 1, wherein, After fusing the first enhancement result and the second enhancement result to obtain the first fusion result, the method further includes: The first fusion result is upsampled to obtain a third feature map, which is used to describe the low-frequency structural features of the target image; A fourth feature map is obtained, which is used to describe the high-frequency detail features of the target image, and the third feature map and the fourth feature map have the same size; The third feature map is subjected to a third convolution process to obtain a third enhancement result; The fourth feature map is subjected to a fourth convolution process to obtain a fourth enhancement result; The third enhancement result and the fourth enhancement result are fused to obtain a second fusion result, which is used to generate an enhanced image of the target image.

3. The method according to claim 1 or 2, characterized in that, The second convolution process on the second feature map to obtain the second enhancement result includes: The second feature map is subjected to a second convolution process based on the conditional feature map to obtain a second enhancement result. The conditional feature map is obtained by feature extraction of the target image through the target network, and the conditional feature map has the same size as the second feature map.

4. The method of claim 3, wherein, The second convolution process on the second feature map based on the conditional feature map includes: The conditional feature map is subjected to an affine transformation SFT to obtain the γ coefficients and β coefficients; The second feature map is subjected to a second convolution process based on the γ coefficient and the β coefficient.

5. The method of claim 2, wherein, The method further includes: In response to the fact that the third feature map has the same size as the target image, the second fusion result is determined to be the enhanced image of the target image.

6. An image processing apparatus characterized by comprising: include: The acquisition unit is used to acquire a first feature map and a second feature map. The first feature map is used to describe the high-frequency detail features of the target image, and the second feature map is used to describe the low-frequency structural features of the target image. The first feature map and the second feature map have the same size. The processing unit is configured to perform a first convolution process on the first feature map to obtain a first enhancement result, wherein the convolutional filter kernel of the first convolution process is used to optimize the tone mapping of the image. The processing unit is further configured to perform a second convolution processing on the second feature map to obtain a second enhancement result. The convolutional filter kernel of the second convolution processing is used to optimize the detailed features of the image, and the convolutional filter kernel of the first convolution processing and the convolutional filter kernel of the second convolution processing are different from each other. The processing unit is further configured to fuse the first enhancement result and the second enhancement result to obtain a first fusion result, which is used to generate an enhanced image of the target image.

7. The image processing apparatus according to claim 6, characterized in that, The processing unit is further configured to upsample the first fusion result to obtain a third feature map, the third feature map being used to describe the low-frequency structural features of the target image; The acquisition unit is further configured to acquire a fourth feature map, which is used to describe the high-frequency detail features of the target image, wherein the third feature map and the fourth feature map have the same size; The processing unit is further configured to perform a third convolution process on the third feature map to obtain a third enhancement result; The processing unit is further configured to perform a fourth convolution process on the fourth feature map to obtain a fourth enhancement result; The processing unit is further configured to fuse the third enhancement result and the fourth enhancement result to obtain a second fusion result, which is used to generate an enhanced image of the target image.

8. The image processing apparatus according to claim 6 or 7, characterized by, The processing unit is specifically used for: The second feature map is subjected to a second convolution process based on the conditional feature map to obtain a second enhancement result. The conditional feature map is obtained by feature extraction of the target image through the target network, and the conditional feature map has the same size as the second feature map.

9. The image processing apparatus according to claim 8, characterized by The processing unit is specifically used for: The conditional feature map is subjected to an affine transformation SFT to obtain the γ coefficients and β coefficients; The second feature map is subjected to a second convolution process based on the γ coefficient and the β coefficient.

10. The image processing apparatus according to claim 7, characterized in that, The image processing device further includes: The determining unit is configured to determine the second fusion result as an enhanced image of the target image when the third feature map has the same size as the target image.

11. A computer device, characterized in that, It includes a processor and a memory, wherein the processor is coupled to the memory. The memory is used to store programs; The processor is configured to execute a program in the memory, causing the computer device to perform the method as described in any one of claims 1 to 5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 5.

13. A computer program product, characterized in that, The computer program product stores computer-readable instructions that, when executed by a processor, implement the method as described in any one of claims 1 to 5.

14. A chip system, characterized in that, The chip system includes at least one processor, which, when program instructions are executed in the at least one processor, causes the method as described in any one of claims 1 to 5 to be executed.

Images