Image processing method and device

Abstract

The present application provides an image processing method and device in the field of artificial intelligence, which enhances the partial features of less concern and introduces multi-scale global features to obtain an output image with better enhancement effect. The method includes: obtaining an input image; obtaining a first feature based on the input image, the first feature is obtained based on at least one local feature extracted from the input image; flipping the values ​​of some features in the first feature to obtain a second feature, the partial feature is a feature of a preset quantity ratio in the first feature; obtaining the global feature of the input image; fusing the second feature and the global feature to obtain the output feature; and obtaining the output image based on the output feature.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a method and device for image processing. Background Art

[0002] In recent years, deep learning methods, especially CNN-based fully convolutional neural network methods, have made great progress in many fields of artificial intelligence and achieved remarkable results in a variety of computer vision underlying tasks.

[0003] In existing image enhancement solutions, local features of an image at multiple scales can be collected, and then the local features at multiple scales can be fused to obtain the final enhanced image. However, because only local features are focused on, the final image is prone to uneven colors, resulting in poor final image quality. Therefore, how to obtain a better enhanced image has become an urgent problem to be solved. Summary of the invention

[0004] The present application provides an image processing method and device, which enhances dark features and introduces multi-scale global features to obtain an output image with better enhancement effect.

[0005] In view of this, in a first aspect, the present application provides a method for image processing, comprising: obtaining an input image; obtaining a first feature based on the input image, the first feature being obtained based on at least one local feature extracted from the input image; flipping the values ​​of some features in the first feature to obtain a second feature, wherein the value before flipping and the value after flipping in the some features are negatively correlated, and the some features are features that account for a preset proportion of the first features; obtaining a global feature corresponding to the input image; fusing the second feature and the global feature to obtain an output feature; and obtaining an output image based on the output feature.

[0006] Therefore, in the implementation mode of the present application, in the process of processing the input image, the values ​​of some features in the first feature are flipped, so that the features that originally received less attention are more prominent, so that the areas that originally received less attention in the final output image perform better, so that the output image is clearer and the enhancement effect is better. The present application proposes image enhancement that focuses on bright and dark areas to obtain a higher dynamic range and better dark performance. Compared with the existing scheme, in the present application, a simple splitting, flipping, and biasing operation is used to obtain the second feature, which effectively distinguishes the bright features from the dark features under the premise of greatly saving the amount of calculation. Then the bright features and the dark features are processed separately, and more attention is paid to the dark features, so that the first neural network has better dark performance and a higher dynamic range. In addition, the second feature and the global feature can be fused. Therefore, in the process of enhancing the image, in addition to paying attention to the local features, the global feature distribution is also paid attention to, so that the feature distribution of the output image is more uniform, avoiding the occurrence of pseudo-color or color unevenness, and obtaining an output image with better enhancement effect.

[0007] In a possible implementation, flipping some features in the first feature to obtain the second feature may include: splitting the first feature according to a preset quantity ratio to obtain a third feature and a fourth feature, where the specific splitting method may be randomly selecting features with a preset quantity ratio from the first feature, or selecting features with a preset quantity ratio from a fixed position in the first feature; flipping the value of the fourth feature to obtain a fifth feature; and fusing the fifth feature with the third feature to obtain the second feature.

[0008] Therefore, in an embodiment of the present application, some features in the first feature are flipped to obtain a fifth feature with a more prominent dark distribution, thereby achieving a better enhancement effect on the dark features in the output image and obtaining a clearer output image.

[0009] In a possible implementation, fusing the fifth feature and the third feature to obtain the second feature may include: performing feature extraction on the fifth feature and the third feature respectively to obtain an enhanced fifth feature and an enhanced third feature; and fusing the enhanced fifth feature and the enhanced third feature to obtain the second feature.

[0010] In the implementation manner of the present application, the fifth feature and the third feature can be further extracted to obtain an enhanced fifth feature and an enhanced third feature, and the enhanced fifth feature and the enhanced third feature are fused to obtain the second feature. Therefore, by enhancing the third feature and the fifth feature, both the dark and bright features of the input image are enhanced, so that the final output image is clearer.

[0011] In a possible implementation, flipping the value of the fourth feature to obtain the fifth feature may include: inverting the value of the fourth feature, and adding a bias value to the inverted value to obtain the fifth feature.

[0012] In the implementation manner of the present application, dark features can be obtained by inverting and increasing the bias, so that the features with lower brightness in the input image are enhanced, and more attention is paid to the dark features, so that the dark part of the output image performs better and a clearer output image is obtained.

[0013] In a possible implementation, the output feature is obtained by taking the input image as the input of the first neural network and outputting it by the first neural network, the first neural network is trained using a training set, the training set includes at least one non-aligned sample, an non-aligned sample includes an input image and a true value image, and the position of the target object included in the input image and the position of the target object included in the true value image are not exactly the same.

[0014] In the implementation manner of the present application, the input image can be enhanced by using a first neural network. The first neural network can be trained using non-aligned samples. Therefore, the image processing method provided by the present application can also be applied to the scenario of non-aligned samples. Based on the first neural network trained with non-aligned samples, the first neural network can also be used to output a clearer image.

[0015] In a possible implementation, before obtaining the first feature according to the input image, the method further includes: downsampling the input image to obtain an updated input image, and the updated input image is used to obtain the first feature.

[0016] In the implementation manner of the present application, local features may be extracted from the downsampled image to obtain the first feature, and processing may be performed on images of different scales.

[0017] In a possible implementation, before obtaining the output image based on the output feature, the above method may also include: upsampling the output feature to obtain an upsampled feature; fusing the first feature and the upsampled feature to obtain an updated first feature, the updated first feature is used to update the second feature, the updated second feature is used to update the output feature, and the updated output feature is obtained.

[0018] In the implementation mode of the present application, the input image can be iteratively processed multiple times to obtain the final output features. Therefore, in the implementation mode of the present application, the input image can be iteratively processed multiple times, and images of different scales can be processed. In each iteration process, global features can be introduced to fuse and obtain output features, so that the final output features are associated with the global feature distribution of the input image, and multi-scale feature fusion is achieved, thereby obtaining a clearer output image.

[0019] In a second aspect, the present application provides an image processing device, comprising:

[0020] An input module, used to obtain an input image;

[0021] A local feature extraction module, used to obtain a first feature according to an input image, where the first feature is obtained according to at least one local feature extracted from the input image;

[0022] An enhancement module, used for flipping the values ​​of some features in the first feature to obtain a second feature, wherein the values ​​of the some features before flipping are negatively correlated with the values ​​after flipping, and the some features are features selected by a preset quantity ratio in the first feature;

[0023] The global feature extraction module is also used to obtain the global features corresponding to the input image;

[0024] A fusion module is used to fuse the second feature and the global feature to obtain the output feature;

[0025] The output module is used to obtain an output image according to the output features.

[0026] In a possible implementation, the enhancement module is specifically used to: split the first feature according to a preset quantity ratio to obtain a third feature and a fourth feature; flip the value of the fourth feature to obtain a fifth feature; and fuse the fifth feature and the third feature to obtain the second feature.

[0027] In a possible implementation, the enhancement module is specifically used to: perform feature extraction on the fifth feature and the third feature respectively to obtain an enhanced fifth feature and an enhanced third feature; and fuse the enhanced fifth feature and the enhanced third feature to obtain a second feature.

[0028] In a possible implementation, the enhancement module is specifically configured to invert the value of the fourth feature, and add a bias value to the inverted value to obtain the fifth feature.

[0029] In a possible implementation, the output feature is obtained by taking the input image as the input of the first neural network and outputting it by the first neural network, the first neural network is trained using a training set, the training set includes at least one non-aligned sample, an non-aligned sample includes an input image and a true value image, and the position of the target object included in the input image and the position of the target object included in the true value image are not exactly the same.

[0030] In a possible implementation, the image processing device further includes: a downsampling module, configured to downsample the input image to obtain an updated input image before the local feature extraction module obtains the first feature based on the input image, and the updated input image is used to obtain the first feature.

[0031] In a possible implementation, the image processing device may further include: an upsampling module, used to upsample the output features to obtain upsampled features before the output module obtains the output image based on the output features; a local feature extraction module, further used to fuse the first feature and the upsampled feature to obtain an updated first feature, the updated first feature is used to update the second feature, the updated second feature is used to update the output feature, and the updated output feature is obtained.

[0032] In a third aspect, the present application provides a neural network training method, including: using samples in a training set as input, iteratively training a first neural network to obtain a trained first neural network, wherein the training set includes one or more non-aligned samples, the non-aligned samples include an input image and a true value image, and the position of the target object included in the input image is not exactly the same as the position of the target object included in the true value image; wherein, in any iterative training process, one or more non-aligned samples are used as input to the first neural network to obtain an output image, and then the difference between the output image and the true value image corresponding to the input image is calculated by a loss function to obtain a loss value, and the first neural network is updated according to the loss value to obtain the first neural network updated output in the current iteration. By using a loss function that tolerates non-pixel-level alignment of training data, problems such as blurring and pseudo-coloring of the output image are avoided, and the output image is ensured to be clear and true.

[0033] Therefore, in the implementation of the present application, non-aligned samples can be used to train the first neural network, and under the premise of ensuring that the output image is clear and realistic, a better output image in terms of color, brightness, contrast, dynamic range, etc. is obtained. Among them, a constraint item that ignores the offset of the target object in the training image is used to weaken the impact of the non-alignment of the training data. At the same time, the constraint item in the feature domain is introduced instead of the traditional pixel domain constraint item to further ensure that the output image is clear and realistic, and obtain an output image with better enhancement effect.

[0034] In a possible implementation, the loss function may include a constraint term that ignores the offset between the output image and the true image, where the offset refers to the offset between the position of the same target object in the output image and the position in the true image. Therefore, in the implementation of the present application, even when training with non-aligned samples, clear and realistic output images can be obtained through training, and the effect is better, it can adapt to more scenes, and the generalization ability is stronger.

[0035] In a fourth aspect, the present application provides a training device, comprising:

[0036] An input module, used for taking samples in the training set as input of the first neural network;

[0037] A training module is used to iteratively train the first neural network to obtain a trained first neural network, wherein the training set includes one or more non-aligned samples, the non-aligned samples include an input image and a true value image, and the position of the target object included in the input image is not completely the same as the position of the target object included in the true value image; wherein, the process of any iterative training includes taking the one or more non-aligned samples as the input of the first neural network to obtain an output image, then calculating the difference between the output image and the true value image corresponding to the input image through a loss function to obtain a loss value, and updating the first neural network according to the loss value to obtain the first neural network with the updated output of the current iteration.

[0038] In a possible implementation, the loss function may include a constraint term that ignores the offset between the output image and the true image, where the offset refers to the offset between the position of the same target object in the output image and the position in the true image. Therefore, in the implementation of the present application, even when training with non-aligned samples, clear and realistic output images can be obtained through training, and the effect is better, it can adapt to more scenes, and the generalization ability is stronger.

[0039] In a fifth aspect, an embodiment of the present application provides an image processing device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the method for image processing shown in any one of the first aspects above. Optionally, the image processing device may be a chip.

[0040] In a sixth aspect, an embodiment of the present application provides a training device, comprising: a processor and a memory, wherein the processor and the memory are interconnected via a line, and the processor calls a program code in the memory to execute the processing-related functions in the neural network training method shown in any one of the third aspects above. Optionally, the training device can be a chip.

[0041] In the seventh aspect, an embodiment of the present application provides an image processing device, which may also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing in the above-mentioned first aspect or any optional embodiment of the first aspect.

[0042] In the eighth aspect, an embodiment of the present application provides a training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform functions related to processing as described in the third aspect or any optional embodiment of the third aspect.

[0043] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium, comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the third aspect.

[0044] In the tenth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute a method in any optional implementation of the first aspect or the third aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A schematic diagram of an artificial intelligence main framework for this application;

[0046] Figure 2 A schematic diagram of a convolutional neural network structure provided in an embodiment of the present application;

[0047] Figure 3 A schematic diagram of a system architecture provided for an embodiment of the present application;

[0048] Figure 4 Another schematic diagram of a system architecture provided for an embodiment of the present application;

[0049] Figure 5 A flowchart of an image processing method provided in an embodiment of the present application;

[0050] Figure 6 A flowchart of another image processing method provided in an embodiment of the present application;

[0051] Figure 7 A flowchart of another image processing method provided in an embodiment of the present application;

[0052] Figure 8A flowchart of another image processing method provided in an embodiment of the present application;

[0053] Fig. 9 A schematic diagram of a flow chart of a feature fusion method provided in an embodiment of the present application;

[0054] Fig.10 A schematic diagram of output image comparison provided in an embodiment of the present application;

[0055] Fig.11 Another output image comparison schematic diagram provided in an embodiment of the present application;

[0056] Fig. 12A A schematic diagram of an input image provided in an embodiment of the present application;

[0057] Fig. 12B A schematic diagram of a true value image provided in an embodiment of the present application;

[0058] Fig.13A Another schematic diagram of a true value image provided in an embodiment of the present application;

[0059] Fig. 13B A schematic diagram of an image output by DPE-Unet;

[0060] Fig. 13C A schematic diagram of an image output by SGN;

[0061] Fig.13D A schematic diagram of an image output by the image processing method provided in this application;

[0062] Fig.14A Another schematic diagram of an input image provided in an embodiment of the present application;

[0063] Fig. 14B Another schematic diagram of a true value image provided in an embodiment of the present application;

[0064] Fig.15A Another schematic diagram of a true value image provided in an embodiment of the present application;

[0065] Fig. 15B This is another image diagram of the DPE-Unet output;

[0066] Fig. 15C A schematic diagram of another image output by SGN;

[0067] Fig.15D A schematic diagram of another image output by the image processing method provided in this application;

[0068] Fig.16A Another schematic diagram of an input image provided in an embodiment of the present application;

[0069] Fig. 16B Another schematic diagram of a true value image provided in an embodiment of the present application;

[0070] Fig.17A Another schematic diagram of a true value image provided in an embodiment of the present application;

[0071] Fig. 17B This is another image diagram of the DPE-Unet output;

[0072] Fig. 17C A schematic diagram of another image output by SGN;

[0073] Fig.17D A schematic diagram of another image output by the image processing method provided in this application;

[0074] Fig.18 A flowchart of a neural network training method provided in an embodiment of the present application;

[0075] Fig.19 A schematic diagram of the structure of a neural network provided in an embodiment of the present application;

[0076] Fig. 20 A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0077] Fig.21 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0078] Fig. 22 A schematic diagram of the structure of another image processing device provided in an embodiment of the present application;

[0079] Fig.23 A schematic diagram of the structure of another training device provided in an embodiment of the present application;

[0080] Fig.24 A schematic diagram of a chip structure provided in an embodiment of the present application. DETAILED DESCRIPTION

[0081] The following will describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0082] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0083] (1) Infrastructure

[0084] Infrastructure provides computing power support for artificial intelligence systems, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, such as central processing units (CPU), neural-network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC) or field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0085] (2) Data

[0086] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0087] (3) Data processing

[0088] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0089] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0090] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0091] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0092] (4) General ability

[0093] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0094] (5) Smart products and industry applications

[0095] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0096] The embodiments of the present application involve a large number of neural network-related applications. In order to better understand the solutions of the embodiments of the present application, the relevant terms and concepts of the neural network that may be involved in the embodiments of the present application are first introduced below.

[0097] (1) Neural Network

[0098] A neural network may be composed of neural units. A neural unit may refer to an operation unit with xs and intercept 1 as input. The output of the operation unit may be as shown in formula (1-1):

[0099]

[0100] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0101] (2) Deep Neural Networks

[0102] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple intermediate layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, intermediate layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all intermediate layers, or hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0103] Although DNN looks complicated, each layer can be expressed as a linear relationship expression: in, is the input vector, is the output vector, is the offset vector or bias parameter, w is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Taking the coefficient w as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0104] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0105] It should be noted that the input layer does not have a W parameter. In a deep neural network, more intermediate layers allow the network to better characterize complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (a weight matrix formed by many layers of vectors W).

[0106] (3) Convolutional Neural Network

[0107] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0108] (4) Recurrent neural networks (RNN), also known as recurrent neural networks, are used to process sequence data. In traditional neural network models, the layers are fully connected from the input layer to the middle layer and then to the output layer, while the nodes within each layer are disconnected. Although this common neural network has solved many difficult problems, it is still powerless to solve many problems. For example, if you want to predict the next word in a sentence, you generally need to use the previous word, because the previous and next words in a sentence are not independent. The reason why RNN is called a recurrent neural network is that the current output of a sequence is also related to the previous output. The specific manifestation is that the network will remember the previous information and apply it to the calculation of the current output, that is, the nodes between the middle layers are no longer disconnected but connected, and the input of the middle layer includes not only the output of the input layer but also the output of the middle layer at the previous moment. In theory, RNN can process sequence data of any length. The training of RNN is the same as the training of traditional CNN or DNN.

[0109] (5) Residual Neural Network (ResNet)

[0110] Residual neural network is proposed as an example to solve the degradation problem caused by too many hidden layers of neural network. The degradation problem means that when the number of hidden layers of the network increases, the accuracy of the network reaches saturation and then degrades sharply. Moreover, this degradation is not caused by overfitting, but when backpropagation is performed, the correlation between the gradients is not large when propagating to the bottom layer, and the gradient update is insufficient, which reduces the accuracy of the predicted label of the final model. When the neural network degenerates, the shallow network can achieve better training results than the deep network. At this time, if the features of the low layer are transmitted to the high layer, the effect should be at least not worse than that of the shallow network. Therefore, this effect can be achieved through an identity mapping. This identity mapping is called a residual connection (shortcut), and it is easier to optimize this residual mapping than to optimize the original mapping.

[0111] It should be understood that the first neural network mentioned in the following embodiments of the present application can be one of the neural networks mentioned above or a combination of multiple neural networks.

[0112] (6) Loss Function

[0113] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0114] (7) Back propagation algorithm

[0115] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0116] (8) Non-aligned samples

[0117] A non-aligned sample may include two images, which include the same target object, but the position of the target object in the two images is not exactly the same. For example, if both image 1 and image 2 include object A, and the position of object A in image 1 and image 2 is different, such as the coordinates of object A in image 1 are (10, 20), and the coordinates in image 2 are (10, 50), image 1 and image 2 are called a non-aligned sample. Generally, a non-aligned sample may include an input image and a ground truth, and the positions of the same object included in the input image and the ground truth may not be exactly the same. Pixel-level alignment means that it does not pay attention to whether the color, brightness and other information of the image are consistent, but only pays attention to whether the objects in the image are aligned. In part or all of the image area, the conditions for pixel-by-pixel alignment are not met, that is, the two images are not pixel-level aligned. Alignment and non-alignment in the following text refer to pixel-level alignment and non-pixel-level alignment, respectively.

[0118] (9) Receptive Field

[0119] A term used in the field of deep neural networks in the field of computer vision to indicate the size of the receptive range of neurons at different positions within the neural network to the original image. The larger the value of the neuron's receptive field, the larger the range of the original image it can access, which also means that the neuron may contain more global and higher-level semantic features; the smaller the value, the more local and detailed the features it contains. The value of the receptive field can be roughly used to determine the level of abstraction of each layer.

[0120] (10) space2depth operation

[0121] A method of data rearrangement, or a downsampling method, the downsampling mentioned in the following embodiments of the present application can be performed in this way. Of course, other methods can also be used for downsampling. For example, for a feature of size [h,w,c], the pixel information of k×k×1 in the h and w directions is rearranged and distributed to the dimension of c (i.e. 1×1×k2 ), resulting in a size of features, changing the pixel arrangement to achieve down sampling.

[0122] (11) depth2space operation

[0123] A way of rearranging data, or a way of upsampling, which corresponds to the inverse operation of space2depth. The upsampling mentioned below in this application can be performed using space2depth or other methods, which can be adjusted according to the actual application scenario. For example, for a feature of size [h,w,c], 1×1×k in the c direction is 2 The pixel information is rearranged and distributed to the dimensions of h and w (i.e., k×k×1), thus generating a features to achieve the upsampling function.

[0124] (12) Image (image quality) enhancement

[0125] It refers to the technology of processing the brightness, color, contrast, saturation, dynamic range, etc. of an image to meet certain specific indicators. Image enhancement and image quality enhancement in this patent describe the same technology.

[0126] Generally, CNN is a commonly used neural network. For ease of understanding, the structure of the convolutional neural network is introduced as an example below.

[0127] Combine the following Figure 2 The structure of CNN is introduced in detail as an example. As mentioned in the basic concept introduction above, convolutional neural network is a deep neural network with convolution structure, which is a deep learning architecture. Deep learning architecture refers to multiple levels of learning at different abstract levels through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network, in which each neuron can respond to the image input into it.

[0128] like Figure 2 As shown, the convolutional neural network (CNN) 200 may include an input layer 210, a convolution layer / pooling layer 220 (wherein the pooling layer is optional), and a neural network layer 230. In the following embodiments of the present application, for ease of understanding, each layer is referred to as a stage. The relevant contents of these layers are described in detail below.

[0129] Convolutional layer / pooling layer 220:

[0130] like Figure 2 The convolution layer / pooling layer 220 shown may include layers 221-226, for example: in one implementation, layer 221 is a convolution layer, layer 222 is a pooling layer, layer 223 is a convolution layer, layer 224 is a pooling layer, layer 225 is a convolution layer, and layer 226 is a pooling layer; in another implementation, layers 221 and 222 are convolution layers, layer 223 is a pooling layer, layer 224 and 225 are convolution layers, and layer 226 is a pooling layer. That is, the output of a convolution layer can be used as the input of a subsequent pooling layer, or as the input of another convolution layer to continue the convolution operation.

[0131] The following will take the convolution layer 221 as an example to introduce the internal working principle of a convolution layer.

[0132] The convolution layer 221 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), that is, multiple isotype matrices, are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolution image, and the dimension here can be understood as being determined by the "multiple" mentioned above. Different weight matrices can be used to extract different features in the image, for example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unwanted noise in the image, etc. The multiple weight matrices have the same size (rows × columns), and the feature maps extracted by the multiple weight matrices of the same size are also the same size. The extracted feature maps of the same size are then merged to form the output of the convolution operation.

[0133] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input image, so that the convolutional neural network 200 can make correct predictions.

[0134] When the convolutional neural network 200 has multiple convolutional layers, the initial convolutional layer (for example, 221) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 200 increases, the features extracted by the later convolutional layers (for example, 226) become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0135] In the following embodiments of the present application, the feature extraction process mentioned may be a process of extracting features through convolution. For example, it may include a separation convolution with a convolution kernel size of 3×3 (sep_conv_3x3), a separation convolution with a convolution kernel size of 5×5 (sep_conv_5x5), a dilated convolution with a convolution kernel size of 3×3 and a dilation rate of 2 (dil_conv_3x3), a dilated convolution with a convolution kernel size of 5×5 and a dilation rate of 2 (dil_conv_5x5), and the like. Moreover, the difference between extracting local features and global features is that local features pay more attention to the features of local areas, while global feature extraction requires combining the global feature distribution in the image to obtain global features.

[0136] Convolutional layer / pooling layer 220:

[0137] Since it is often necessary to reduce the number of training parameters, it is often necessary to periodically introduce a pooling layer after the convolution layer. The pooling layer can also be called a downsampling layer, which can be used to downsample the image. Figure 2 Each layer 221-226 illustrated in 220 may be a convolution layer followed by a pooling layer, or may be multiple convolution layers followed by one or more pooling layers. In the image processing process, the purpose of the pooling layer is to reduce the spatial size of the image. The pooling layer may include an average pooling operator and / or a maximum pooling operator for sampling the input image to obtain an image of smaller size. The average pooling operator may calculate the pixel values ​​in the image within a specific range to generate an average value as the result of average pooling. The maximum pooling operator may take the pixel with the largest value in the range within a specific range as the result of maximum pooling. In addition, just as the size of the weight matrix used in the convolution layer should be related to the image size, the operator in the pooling layer should also be related to the image size. The size of the image output after processing by the pooling layer may be smaller than the size of the image input to the pooling layer, and each pixel in the image output by the pooling layer represents the average value or maximum value of the corresponding sub-region of the image input to the pooling layer.

[0138] Neural Network Layer 230:

[0139] After being processed by the convolution layer / pooling layer 220, the convolution neural network 200 is not sufficient to output the required output information. As mentioned above, the convolution layer / pooling layer 220 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 200 needs to use the neural network layer 230 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 230 may include multiple intermediate layers (such as Figure 2 231, 232 to 23n) and an output layer 240, which may also be referred to as a fully connected (FC) layer, the parameters contained in the multiple intermediate layers may be pre-trained according to relevant training data of a specific task type, for example, the task type may include image enhancement, image recognition, image classification, image super-resolution reconstruction, etc.

[0140] After the multiple intermediate layers in the neural network layer 230, that is, the last layer of the entire convolutional neural network 200 is the output layer 240, which has a loss function similar to the classification cross entropy, specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 200 (such as Figure 2 The propagation from 210 to 240 is forward propagation), and the reverse propagation (such as Figure 2 The propagation from 240 to 210 is called back propagation) and then starts to update the weight values ​​and biases of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the result output by the convolutional neural network 200 through the output layer and the ideal result.

[0141] It should be noted that if Figure 2 The convolutional neural network 200 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network may also exist in the form of other network models.

[0142] In this application, you can use Figure 2 The convolutional neural network 200 shown in FIG. 1 processes the image to be processed to obtain an enhanced image. Figure 2 As shown, after the image to be processed is processed by the input layer 210, the convolution layer / pooling layer 220 and the neural network layer 230, the output is enhanced to be a clearer image containing more texture information.

[0143] The method for image processing provided in the embodiment of the present application can be executed on a server or on a terminal device, or the first neural network mentioned below in the present application can be deployed on a server or on a terminal, and can be adjusted according to the actual application scenario. The terminal device can be a mobile phone with image processing function, a tablet personal computer (tablet personal computer, TPC), a media player, a smart TV, a laptop computer (laptop computer, LC), a personal digital assistant (personal digital assistant, PDA), a personal computer (personal computer, PC), a camera, a video camera, a smart watch, a wearable device (wearable device, WD) or an autonomous driving vehicle, etc., and the embodiment of the present application does not limit this.

[0144] like Figure 3 As shown, the embodiment of the present application provides a system architecture 100. Figure 3 In the embodiment, the data acquisition device 160 is used to acquire training data. In some optional implementations, for image enhancement, the samples included in the training data may be aligned samples or non-aligned samples, and each sample may be two images with different sharpness. Aligned samples may be understood as objects included in two images having exactly the same positions in the two images, and non-aligned images may be understood as objects included in two images having different positions in the two images.

[0145] After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 obtains the target model / rule 101 through training based on the training data maintained in the database 130. Optionally, the training set mentioned in the following embodiments of the present application can be obtained from the database 130 or through user input data.

[0146] Among them, the target model / rule 101 can be the first neural network after training in the embodiment of the present application.

[0147] The following describes how the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input original image and compares the output image with the original image until the difference between the image output by the training device 120 and the original image is less than a certain threshold, thereby completing the training of the target model / rule 101.

[0148] The above-mentioned target model / rule 101 can be used to implement the first neural network trained by the method for image processing of the embodiment of the present application, that is, the data to be processed (such as an image) is input into the target model / rule 101 after relevant preprocessing, and the processing result can be obtained. The target model / rule 101 in the embodiment of the present application can specifically be the first neural network mentioned below in the present application, and the first neural network can be a neural network of the aforementioned CNN, DNN or RNN type. It should be noted that in actual applications, the training data maintained in the database 130 does not necessarily all come from the collection of the data acquisition device 160, and may also be received from other devices. It should also be noted that the training device 120 does not necessarily train the target model / rule 101 completely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training. The above description should not be used as a limitation on the embodiment of the present application.

[0149] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 3 The execution device 110 shown in the figure may also be called a computing device. The execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, an augmented reality (AR) / virtual reality (VR), a vehicle terminal, etc., or a server or a cloud device, etc. Figure 5 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include: data to be processed input by the client device in the embodiment of the application.

[0150] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to input data (such as data to be processed) received by I / O interface 112. In an embodiment of the present application, preprocessing module 113 and preprocessing module 114 may not be provided (or only one of the preprocessing modules may be provided), and computing module 111 may be used directly to process the input data.

[0151] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0152] Finally, the I / O interface 112 returns the processing result to the client device 140 for providing it to the user. For example, if the first neural network is used for image classification and the processing result is a classification result, the I / O interface 112 returns the classification result obtained above to the client device 140 for providing it to the user.

[0153] It should be noted that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks, and the corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. In some scenarios, the execution device 110 and the training device 120 can be the same device, or located in the same computing device. For ease of understanding, this application will introduce the execution device and the training device separately, without limitation.

[0154] exist Figure 3 In the case shown, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data input to the I / O interface 112 as shown in the figure and the predicted label output from the I / O interface 112 as new sample data, and store it in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data input to the I / O interface 112 as shown in the figure and the predicted label output from the I / O interface 112 as new sample data in the database 130.

[0155] It should be noted that Figure 3 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0156] like Figure 3As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 in the embodiment of the present application can be the first neural network in the present application. Specifically, the first neural network provided in the embodiment of the present application can be CNN, deep convolutional neural networks (DCNN), recurrent neural networks (RNN), etc.

[0157] See attached Figure 4 , the embodiment of the present application also provides a system architecture 400. The execution device 110 is implemented by one or more servers, and optionally cooperates with other computing devices, such as data storage, routers, load balancers, etc.; the execution device 110 can be arranged on a physical site, or distributed on multiple physical sites. The execution device 110 can use the data in the data storage system 150, or call the program code in the data storage system 150 to implement the following of the present application Figure 5 Corresponding steps of the method for image processing.

[0158] Users can operate their respective user devices (e.g., local device 401 and local device 402) to interact with execution device 110. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0159] The local device of each user can interact with the execution device 110 through a communication network of any communication mechanism / communication standard, and the communication network can be a wide area network, a local area network, a point-to-point connection, or any combination thereof. Specifically, the communication network may include a wireless network, a wired network, or a combination of a wireless network and a wired network. The wireless network includes, but is not limited to: a fifth generation mobile communication technology (5th-Generation, 5G) system, a long term evolution (long termevolution, LTE) system, a global system for mobile communication (global system for mobile communication, GSM) or a code division multiple access (code division multiple access, CDMA) network, a wideband code division multiple access (wideband code division multiple access, WCDMA) network, wireless fidelity (wireless fidelity, WiFi), Bluetooth (bluetooth), Zigbee protocol (Zigbee), radio frequency identification technology (radio frequency identification, RFID), long range (Lora) wireless communication, near field communication (near field communication, NFC) in any one or more combinations. The wired network may include a fiber optic communication network or a network composed of coaxial cables, etc.

[0160] In another implementation, one or more aspects of the execution device 110 may be implemented by each local device, for example, the local device 401 may provide local data or feedback calculation results to the execution device 110. The local device may also be referred to as a computing device.

[0161] It should be noted that all functions of the execution device 110 may also be implemented by a local device. For example, the local device 401 implements the functions of the execution device 110 and provides services to its own user, or provides services to the user of the local device 402.

[0162] Deep learning methods, especially the fully convolutional neural network method based on CNN, have been developed in many fields of artificial intelligence and have achieved remarkable results in a variety of computer vision underlying tasks. In image classification and image quality enhancement tasks, methods based on deep learning have surpassed traditional methods. For example, the proposed VGG classifier has a significantly better accuracy than traditional image classifiers. However, with the development of display technology, the requirements for image enhancement effects are higher. Therefore, the present application provides an image processing method for achieving better image enhancement, obtaining clearer images, and improving user experience.

[0163] The following is a detailed introduction to the image processing method provided by this application in combination with the aforementioned scenarios and system architecture.

[0164] Furthermore, the image processing method provided in the present application can be implemented by a first neural network. For example, the first neural network can be deployed on a user's terminal, and the terminal can run the first neural network, thereby implementing the steps of the image processing method provided in the present application.

[0165] First, see Figure 5 , a flow chart of an image processing method provided in the present application is as follows.

[0166] 501. Obtain an input image.

[0167] Among them, the input image is the image that needs to be enhanced,

[0168] The input image may be obtained by user input, for example, see Figure 3 , the user can send input data to the execution device 110 through the client device 140, and the input data can carry the input image.

[0169] 502. Process the input image at least once to obtain output features.

[0170] Specifically, after obtaining the input image, the input image may be enhanced once or iteratively enhanced multiple times to obtain enhanced output features.

[0171] More specifically, any processing process in step 502 may specifically include the following steps 5021-5024, as described below.

[0172] 5021. Obtain a first feature according to the input image.

[0173] After acquiring the input image, features can be extracted based on the input image to obtain a first feature. The first feature is obtained based on at least one local feature in the input image, such as extracting features from the at least one local feature or fusing the at least one local feature with other features to obtain the first feature, or the first feature includes at least one local feature of the input image.

[0174] The at least one local feature may be extracted from the input image or the processed input image in units of receptive fields. For example, feature extraction may be performed directly from the input image in units of receptive fields, thereby obtaining multiple local features of the input image to form the first feature. Alternatively, after downsampling the input image, feature extraction may be performed on the downsampled image or feature in units of receptive fields to obtain the first feature. The receptive field may be of a certain size and may be obtained by calculation, such as the calculation may be expressed as: Among them, l k represents the receptive field size of the kth convolution, f k is the kernel size of the kth convolution, s k is the stride of the kth convolution.

[0175] In one scenario, if the input image is processed once to obtain output features, then in step 5021, local features can be directly extracted from the input image using the receptive field as a unit to obtain the first feature.

[0176] In another scenario, if the input image is iteratively processed multiple times, the processing process of the input image may be different in different iterative processes. For example, in one of the iterative processes, if the current process is the first process, the input image may be downsampled once or multiple times, and then local features may be extracted from the downsampled image or features to obtain the first feature; if the current process is the last iterative process, local features may be directly extracted from the input image to obtain the first feature, and the output features obtained in the previous iterative process may be upsampled to obtain the upsampled features, and the first features extracted from the input image and the upsampled features may be fused to obtain the updated first feature; if the current process is not the first or last iterative process in multiple iterative processes, the input image may be downsampled, and then local features may be extracted from the downsampled image to obtain the first feature, and the output features obtained in the previous iterative process may be upsampled to obtain the upsampled features, and the first features extracted from the downsampled image and the upsampled features may be fused to obtain the updated first feature.

[0177] For example, take the input image as an example and process it three times. Figure 6The processing of scale 3, scale 2 and scale 1 shown in . In the processing of scale 1, the input image is downsampled twice, and then multiple local features are extracted from the images obtained by the two downsamplings to obtain the first feature, and then the first feature is processed in other ways. The other processing here refers to the following steps 5022-5024, which will not be repeated here, to obtain the output feature in the processing of scale 1, and then the output feature is upsampled to obtain the upsampled feature. In the processing of scale 2, the input image is downsampled once, and then the image obtained by the one downsampling is subjected to feature extraction, and multiple local features are extracted to form the first feature of scale 2, and then the first feature of scale 2 and the upsampled feature obtained in the processing of scale 1 are fused to obtain the first feature after the update of scale 2. Then the updated first feature is processed in other ways to obtain the output feature, and the output feature is upsampled to obtain the corresponding upsampled feature in the process of scale 2. In the processing process of scale 3, multiple local features can be directly extracted from the input image to form a first feature, and then the first feature of scale 3 and the upsampled feature obtained in the processing process of scale 2 are combined to obtain the first feature after scale 3 update. After other processing is performed on the updated first feature, the final output feature can be obtained.

[0178] 5022. Flip the values ​​of some features in the first feature to obtain the second feature.

[0179] In the process of flipping some features in the first feature, the values ​​of the partial features may be transformed linearly or nonlinearly so that the values ​​of the partial features before flipping are negatively correlated with the values ​​after flipping.

[0180] The proportion of the partial feature to the first feature is a preset number ratio. The partial feature can be a feature selected from a random position in the first feature, or a feature selected from a pre-set position in the first feature. It can be adjusted according to the actual application scenario, and is not limited here. For example, the first feature can be represented as a vector, and 20% of the values ​​can be randomly selected from the first feature for flipping, or the position of the feature to be flipped can be preset, such as setting the first 20% of the values ​​to be selected from the first feature for flipping. In addition, if the input image is iteratively optimized for multiple times, the position of the partial feature selected from the first feature is the same during each iterative optimization process. For example, in the first iterative optimization process of the input image, the first 1 / 2 of the features in the first feature are selected for flipping, then in the second or more iterative optimization processes of the input image, the first 1 / 2 of the features in the first feature are selected for flipping, so that the positions of the features flipped in each iteration process are the same in the input image.

[0181] In a possible implementation, after obtaining the first feature, the third feature and the fourth feature are obtained from the first feature. For ease of understanding, the third feature here can also be called a bright feature. The value of the fourth feature is flipped, and it is converted from a bright feature to a dark feature to obtain a fifth feature. For ease of understanding, the fifth feature is hereinafter called a dark feature. Thereafter, the third feature and the fourth feature are further extracted and enhanced to obtain the second feature.

[0182] For example, the first feature can be represented as F, and F is split into two parts, namely the third feature and the fourth feature. For ease of understanding, the third feature can be represented as F 1 , the fourth feature is expressed as F 2 .

[0183] For example, a feature can be represented as a 4-dimensional vector of [N, H, W, C], where N represents the batch size, H and W represent the size of the feature, and C represents the number of channels of the feature. Slicing can be used to split the feature from the channel dimension, such as splitting 4 channels into two sets of features with 2 channels each. For example, the first feature can be represented as [n, h, w, 2c], that is, the number of channels is 2c, and the split features can be represented as follows: The third feature F 1 =F[:,:,:,0:c]; fourth feature F 2 =F[:,:,:,c:2c]. Therefore, in the embodiment of the present application, the third feature F 1 With the fourth feature F 2 From the same input feature F, therefore, F 1 With F 2 There are the same feature extraction steps, and there is no need to extract the two features separately, which greatly reduces the amount of calculation.

[0184] In addition, the channel dimension of the first feature may be changed through a convolution operation, and then the first feature with the changed channel dimension may be split.

[0185] Optionally, before feature fusion, the third feature may be enhanced to obtain an enhanced third feature, and the fifth feature may be processed to obtain an enhanced fifth feature; and then the enhanced third feature and the enhanced fifth feature may be fused to obtain the second feature.

[0186] The dark feature and the bright feature are enhanced by performing a convolution operation on the dark feature and the bright feature. For example, a convolution operation with a convolution kernel size of 3 and a step size of 1 can be used to enhance the dark feature and the bright feature, respectively, to obtain an enhanced third feature and an enhanced fifth feature.

[0187] Among them, the method of flipping the fourth feature obtained from the first feature may include linearly changing or nonlinearly transforming the values ​​of some features in the first feature, such as negating or adding a bias after negation, etc., which can be adjusted according to the actual application scenario, and the values ​​of some features in the first feature before flipping are negatively correlated with the values ​​after flipping. For example, taking adding a bias after negation as an example, the value of the fourth feature can be negated, and the bias value can be added to the negated value to obtain the fifth feature. For example, the fourth feature can be expressed as F 2 , negate it to get -F 2 , then increase the bias value α to get the fifth feature F 3 =α-F 2 . The value of α corresponding to each channel may be different, and the range of α can usually be set within a certain range. Usually, the data distribution characteristics of each channel may be different, so the value of α corresponding to each channel may be different. For example, the value of α can be set in the range of [1-∈, 1+∈]. For example, ∈=0.1, α can be expressed as α=[α0,α1,…,α c-1 ],α i ∈[0.9,1.1].

[0188] Among them, the method of fusing features can be implemented by using the method of "concatenation + convolution". For example, the two features to be fused can be spliced ​​in the channel dimension, such as splicing the enhanced third feature and the enhanced fifth feature in the channel dimension, and then using the convolution operation with a convolution kernel size of 3 and a step size of 1 to fuse the enhanced third feature and the enhanced fifth feature to obtain the second feature.

[0189] Usually, since the pixel values ​​of the areas with higher brightness are generally larger, their influence in the loss function is also greater, and the constraints of the loss function are also stronger, while the pixel values ​​of the areas with lower brightness are generally smaller, and their influence in the loss function is smaller, and the constraints of the loss function are also weaker, which to a certain extent leads to the enhancement effect of the areas with lower brightness being worse than that of the areas with higher brightness. Therefore, in the embodiments of the present application, by flipping some features, the dark features with originally smaller eigenvalues ​​are made to have larger eigenvalues, and more attention is paid to the dark features, which is beneficial to improving the performance of the first neural network in the dark area, making the dark features in the obtained light and dark features more concerned, so that some originally unclear textures in the input image can be made clearer, thereby improving the clarity of the final output image and achieving a better image enhancement effect.

[0190] 5023. Obtain global features corresponding to the input image.

[0191] Among them, the global features of the input image can be extracted by feature extraction. Specifically, the global features can be extracted from the input image, or extracted from the first feature, or extracted from the second feature, etc., which can be adjusted according to the actual application scenario. For example, the present application uses the extraction of the global features from the second feature as an example for illustrative explanation, which is not intended to be limiting.

[0192] Specifically, the second feature can be globally averaged pooled in width and height dimensions, and the global feature can be output through a fully connected layer. For example, the size of the second feature can be represented by [n, h, w, c], and the second feature is averaged in width and height dimensions to obtain an average value of size [n, 1, 1, c], and then the average value is used as the input of the fully connected layer to output the global feature.

[0193] It should be noted that if the global feature is extracted from the first feature, the present application does not limit the execution order of steps 5022 and 5023. Step 5022 may be executed first, or step 5023 may be executed first, or steps 5022 and 5023 may be executed simultaneously, which may be adjusted according to the actual application scenario. If the global feature is extracted from the input image, the present application does not limit the execution order of steps 5021 and 5023. Step 5021 may be executed first, or step 5023 may be executed first, or steps 5021 and 5023 may be executed simultaneously, which may be adjusted according to the actual application scenario.

[0194] 5024. The second feature and the global feature are integrated to obtain the output feature.

[0195] Specifically, after obtaining the second feature and the global feature, the second feature and the global feature may be concatenated to obtain a concatenated feature, and then the concatenated feature may be fused through a convolution operation to obtain a final output feature.

[0196] More specifically, the global feature can be concatenated from the channel dimension to the second feature to obtain a concatenated feature, and then a convolution operation is performed on the concatenated feature to obtain the final output feature.

[0197] In addition, if the input image is iteratively processed for multiple times, and the current processing is not the last iteration, the output features can also be upsampled to obtain the upsampled features, so that the upsampled features can be used for the next iteration to obtain the final output features. After the last iterative fusion of the input image, the final output features are output.

[0198] Therefore, in the embodiments of the present application, in addition to extracting clearer and more accurate local features, global features are also extracted at each scale of the network, and local features and global features are fused at each scale. Usually, local features only focus on local features, while global features focus more on global feature distribution. Therefore, by fusing local features and global features, the accuracy of the final output features is higher than that of local features and global features, thereby obtaining more accurate output features and achieving more effective enhancement of the final image. Moreover, the first neural network can be divided into processing processes of multiple scales, and different scales of the input image are processed respectively, and extraction and fusion operations of global information are introduced at multiple scales respectively, so that the first neural network can fuse accurate global information at each scale, which is conducive to the network learning more accurate and uniform color and brightness information.

[0199] 503. Obtain an output image according to the output features.

[0200] Among them, the output features usually have multiple channels and cannot be directly displayed in the display interface, so it is necessary to convert the output features into a displayable output image.

[0201] For example, if the output feature has 6 channels, it is necessary to convert the 6 channels into a displayable 3-channel image through a convolution operation to obtain the final output image and enhance the input image.

[0202] Therefore, in the implementation of the present application, some of the less concerned features in the input image can be flipped, so that the dark features that are less concerned are enhanced, so that the final output image can pay more attention to the dark features in the input image, so that the final output image is clearer, and the input image is enhanced. In addition, in the implementation of the present application, the feature extraction of images of different scales can be realized by multiple iterative processing of the input image, so as to combine the features extracted from multiple scales to obtain an output image with richer details and textures. In addition, the present application can perform multiple iterative processing on the input image to obtain multi-scale global information to obtain more accurate color and brightness information. Specifically, for the multi-scale image enhancement network, the present application introduces the extraction and fusion of global information at each scale, so that each scale can accurately fuse the global information of each scale, so that the model obtains more accurate color and brightness information. The present application proposes image enhancement that focuses on bright and dark areas to obtain a higher dynamic range and better dark performance. Compared with the existing scheme, in the present application, a simple splitting, flipping, and biasing operation is used to obtain the second feature, which effectively distinguishes bright features from dark features while greatly saving the amount of calculation. The second feature is then processed separately, giving more attention to the dark feature, so that the first neural network has better dark performance and a higher dynamic range.

[0203] The above-mentioned process of the image processing method provided by the present application is described in detail. Figure 5 The process in the embodiment is combined with specific application scenarios to provide a more detailed description of the image processing method provided in this application.

[0204] For example, see Figure 7 , taking the example of performing three iterative processing on the input image as an example for exemplary explanation.

[0205] Among them, each iteration process can be understood as the process of processing images of different scales. Figure 7 The three iterations described in the figure may be iterative step 701 for scale 1, step 702 for scale 2, and step 703 for scale 3, respectively. Steps 701-703 are respectively described in detail below.

[0206] Step 701, processing the image of scale 1.

[0207] After obtaining the input image, the input image is downsampled twice to obtain an image or feature smaller than the input image. For example, for an image with a size of H*W, it is downsampled by 2 times, that is, an image with a resolution of (H / 2)*(W / 2) is obtained. Alternatively, features with a size of the target resolution can be directly extracted by convolution operation.

[0208] For example, in the embodiment of the present application, a feature with a size of (H / 2)*(W / 2) can be obtained by space2depth, as expressed as The subscript 1 indicates scale 1.

[0209] In step 701, since it is the first time to process the input image, scale fusion is not required, and the obtained That is, after the first feature mentioned above, the feature Brightness and darkness are enhanced to obtain the second feature.

[0210] Alternatively, if downsampling is performed through a convolution operation, a feature of size (H / 2)*(W / 2) can be obtained, and the feature (i.e., the first feature mentioned above) can be directly enhanced to obtain a light and dark feature, i.e., the second feature.

[0211] For example, the brightness enhancement process can be performed as follows: Figure 8 As shown, the features output after downsampling 701 are (801) is split into 802 to obtain and in Bright feature After the light-dark conversion, it is transformed into dark features

[0212] Then, the dark part feature is converted into bright and dark, i.e., the dark and bright conversion 803. For example, for the feature Negate Then increase the bias value α to get the dark feature:

[0213] Then, the bright features Perform feature enhancement 805 and dark features Perform feature enhancement 804 to obtain enhanced bright features With enhanced dark features

[0214] Then, the enhanced bright features With enhanced dark features Perform feature fusion 806, which can be specifically to enhance the bright feature With enhanced dark features The spliced ​​features are then concatenated and fused through a convolution operation to obtain light and dark features 807.

[0215] After performing the light and dark enhancement process 7013 to obtain the light and dark features, a global feature extraction 7014 is performed on the light and dark features to obtain a global feature. The global feature may also be extracted from the input image, or may be extracted from the image obtained after downsampling 7011. Here, the extraction of the global feature from the light and dark features is only used as an example for illustrative description, and is not intended to be limiting.

[0216] It should be noted that the input features of global feature extraction 7014 and brightness enhancement processing 7013 are extracted in different ways. The input features of brightness enhancement processing 7013 are extracted by extracting local features, that is, features extracted from the downsampled image in units of a preset size. The global feature extraction 7014 can be to average the brightness features [n,h,w,c] to obtain [n,1,1,c], and then output the global features through a full connection operation.

[0217] After obtaining the global features, the global features and the light and dark features can be fused to obtain the output features of the first processing, and the output features are upsampled 7016, and the upsampled features are used as inputs of the second iterative processing 702.

[0218] During the second iterative processing 702 of the input image, the process is similar to step 701, such as steps 7021, 7023-7026 are similar to steps 7011, 7013-7016, and similar steps are not repeated here. The difference is that after downsampling 7021, feature extraction is performed on the downsampled image, and then the extracted features and the upsampled features obtained in step 701 are scaled fused 7022 to obtain new features.

[0219] For example, the fusion process can be as follows Fig. 9 As shown, after getting the up-sampled features and local features Afterwards, the upsampled features are fused and local features The downsampled features can be expressed as [n, h, w, c1], and the upsampled features can be expressed as [n, h, w, c2]. The upsampled features and the downsampled features are concatenated to obtain [n, h, w, c1 + c2]. Then, the convolution operation with a convolution kernel of 1*1 and a step size of s = 1 is used to process [n, h, w, c1 + c2] to obtain the features of [n, h, w, c], as expressed as

[0220] Step 703 is similar to step 702, and steps 7032-7035 are similar to steps 7022-7025. Similar steps will not be repeated here. The difference is that there is no need to downsample, and the first feature can be directly extracted from the input graph, and after global fusion is performed to obtain a new fused feature, the new feature can be directly used as the output feature without upsampling.

[0221] Typically, the aforementioned steps 701-703 can be implemented by a trained neural network. For example, a first neural network can be deployed on a user's terminal, and then the terminal runs the first neural network to enhance the image.

[0222] Generally speaking, learning methods for neural networks can be divided into two categories: supervised learning and unsupervised learning.

[0223] Supervised learning has a good performance in low-level image enhancement tasks, but supervised learning requires aligned training data. Aligned training data is very difficult to obtain, which seriously limits the development of supervised learning in the field of low-level image enhancement. Using non-aligned data and training with supervised learning methods will cause blurring and loss of details. Fig.10The supervised image quality enhancement shown in the figure is an example. The left side is the original input image, the middle is the result of using supervised learning methods and non-aligned training data, and the right side is the true value image. It can be clearly seen from the figure that using supervised learning methods and using non-aligned data to train the network will cause serious blurring and detail loss problems.

[0224] Unsupervised learning methods can avoid blurring and loss of details caused by misaligned training data, but unsupervised learning generally faces problems such as unstable model training and easy generation of false colors and artifacts. Fig.11 Take the unsupervised image quality enhancement shown in Figure 1 as an example. The left side is the original input image, the middle is the unsupervised learning enhanced image, and the right side is the true value image. It can be clearly seen from the figure that the unsupervised method produces obvious artifacts.

[0225] In the implementation manner of the present application, non-aligned data can be used to train the first neural network, and multiple constraints are added to the loss function. For example, the loss function can be expressed as L = L content +α*L color +β*L visual +ρ*L shift , so that the constraints of features, colors, perception or ignoring offsets are used as constraints for updating the first neural network, so that the first neural network can output an image with better enhancement effect.

[0226] For ease of understanding, the enhanced image obtained by the image processing method provided in the present application is exemplarily described below in conjunction with a more specific image.

[0227] The image quality can be measured by peak signal-to-noise ratio (PSNR), structural similarity (SSIM) or natural image quality evaluation (NIQE). For example, a self-guided network (SGN) and DPE-Unet are selected for comparison. DPE-Unet is a neural network similar to a U-shaped structure.

[0228] On a given training set and test set, the embodiment of this patent is compared with the following benchmark algorithms, and the specific performance is shown in Table 1.

[0229]

[0230] Table 1

[0231] Among them, Input and GT are included in a non-aligned sample.

[0232] Usually, PSNR is the peak signal-to-noise ratio that characterizes an image, and SSIM is an indicator that characterizes the structural similarity of an image. Generally speaking, the larger the value, the closer the test image is to the real image. However, PSNR / SSIM is generally used as an indicator to measure the quality of aligned images. For non-aligned images, the PSNR / SSIM indicator can only be used as a reference; NIQE is an aesthetic evaluation of the test image, which does not require a target image (true value image) and can directly evaluate the aesthetic quality of the test image. Generally speaking, the smaller the value, the higher the aesthetic quality of the test image. Since the calculation of NIQE does not require an aligned target image (true value image), NIQE is a more accurate evaluation indicator for non-aligned images.

[0233] Therefore, it is obvious that the first neural network provided in the present application can achieve better output. When non-aligned samples are used for training, its output result is better. When the test set is used for testing, the NIQE of the image output by the first neural network provided in the present application is closer to the NIQE of GT.

[0234] In addition, the output results of the first neural network provided in the present application not only greatly exceed those of DPE-UNet and SGN in terms of numerical indicators (PSNR / SSIM), but the output images are also better than those of DPE-UNet and SGN in terms of display effect.

[0235] For example, the input image of the non-aligned sample, such as Fig. 12A As shown, the GT image is Fig. 12B As shown, one of the regions is selected from the input image for comparison, such as Fig. 12B As shown in 1201. The GT image corresponding to area 1201 is as follows Fig.13A As shown in , its NIQE value is 13.2098. Correspondingly, the image output by DPE-UNet is as follows Fig. 13B As shown, the image output by SGN is as follows Fig. 13C As shown, the output result of the first neural network provided by this application is as follows Fig.13D Obviously, the image provided by this application has brighter colors and clearer textures, and the effect shown is significantly better than the images output by DPE-UNet and SGN.

[0236] Also exemplarily, as shown in the input image of the non-aligned sample, Fig.14A As shown, the GT image is Fig. 14B As shown, one of the regions is selected from the input image for comparison, such as Fig. 14B As shown in 1401. The GT image corresponding to region 1401 is as follows Fig.15AAs shown in , its NIQE value is 16.6024. Correspondingly, the image output by DPE-UNet is as follows Fig. 15B As shown, the image output by SGN is as follows Fig. 15C As shown, the output result of the first neural network provided by this application is as follows Fig.15D Obviously, the image provided by this application has brighter colors and clearer textures, and the effect shown is significantly better than the images output by DPE-UNet and SGN.

[0237] Also exemplarily, as shown in the input image of the non-aligned sample, Fig.16A As shown, the GT image is Fig. 16B As shown, one of the regions is selected from the input image for comparison, such as Fig. 16B As shown in 1601. The GT image corresponding to area 1601 is as follows Fig.17A As shown in , its NIQE value is 16.6024. Correspondingly, the image output by DPE-UNet is as follows Fig. 17B As shown, the image output by SGN is as follows Fig. 17C As shown, the output result of the first neural network provided by this application is as follows Fig.17D Obviously, the image provided by this application has brighter colors and clearer textures, and the effect shown is significantly better than the images output by DPE-UNet and SGN.

[0238] Therefore, in the present application, an image enhancement method that focuses on light and dark information is proposed, giving more attention to dark areas. Simple splitting, flipping, and biasing operations are used to effectively distinguish light and dark features. After enhancing the light features and dark features respectively, the light features and dark features are fused to form high-dimensional features that focus on light and dark areas. Effectively improve the dynamic range and dark area performance of the network. Multi-scale global information is proposed. In the multi-scale first neural network, the extraction and fusion of global information are explicitly introduced at each scale to obtain more accurate color domain brightness information. In addition, the first neural network provided in the present application can tolerate the non-alignment of training data, so that the network can learn clear textures and accurate colors in non-aligned data sets, while retaining the details of the input image without blurring.

[0239] In a possible implementation, the image processing method provided by the present application can be implemented by a first neural network, that is, the aforementioned steps 501-503 can be steps performed by the first neural network, and the input image is used as the input of the first neural network. After the first neural network performs steps 501-503, the final output image is output, thereby enhancing the input image. In particular, when using a training set to obtain the first neural network, a training set including non-aligned samples can be used, that is, the network can output clear, high-quality enhanced images under the condition of non-aligned training data.

[0240] The present application also provides a neural network training method, which can use a training set including non-aligned samples to train an initial first neural network to obtain a trained first neural network. Even in the scenario of non-aligned data, a trained model can be obtained to output a clearer image.

[0241] The process of the neural network training method provided in this application can be as follows: Fig.18 shown.

[0242] First, one or more non-aligned samples are included in the training set 1801. Of course, the neural network training method provided in the present application can also be trained using a training set including aligned samples, which can be adjusted according to the actual application scenario.

[0243] The samples in the training set 1801 are used as the input of the first neural network 1802, thereby obtaining the output image of the first neural network 1802. The process of obtaining the output image by the first neural network can refer to the aforementioned Figure 5-Figure 10 The relevant description in will not be repeated here.

[0244] Then, the output image and the true value image of the input sample are used as inputs of the loss function 1803, and the loss value of the loss function 1803 is output. It can be understood that the difference between the output image and the true value image in the input sample can be calculated by the loss function.

[0245] Subsequently, if the loss value output by the loss function is not within a preset range, or is not less than a preset value, etc., the loss value is used to update the parameters of the first neural network, such as weight parameters or structural parameters, etc., to obtain the first neural network of the current iteration.

[0246] Among them, in the process of each iterative update of the first neural network, it can be determined whether the first neural network obtained by each iterative update meets the preset conditions, such as if the loss value is less than the preset value, the difference between the loss value of the current iteration and the loss value outputted in the previous iteration is less than the preset value, or the number of iterations reaches the preset number, then the update of the first neural network can be stopped and the trained first neural network can be output.

[0247] More specifically, the loss function used when updating the first neural network includes multiple constraints, such as constraints on features, constraints on colors, constraints on visual perception, and constraints that can ignore the offset of non-aligned samples, etc. For example, in the process of updating the first neural network, multiple indicators such as color correction and feature similarity can be used to constrain the first neural network, thereby obtaining a model with better image enhancement effect.

[0248] Such as loss function: L = L content +α*L color +β*L visual +ρ*L shift .

[0249] Among them, L content =L CoBi (enhanced, GT), enhanced represents the output of the first neural network, and GT represents the true value image of the non-aligned sample. In the implementation of the present application, by extracting high-dimensional features from the output image and the non-aligned true value image, and optimizing the similarity between the high-dimensional features of the output image and the high-dimensional features of the true value image, the problem of blurred output images when the training data is non-aligned is effectively solved, and image fidelity is achieved.

[0250] Color Constraint: L color =(Blur(enhanced)-Blur(GT)) 2 , α represents the constraint strength of the color constraint item, which can usually be a fixed value and can be adjusted according to the actual application scenario. Blur is a blur operation, and the specific blur method is not limited. In this embodiment, a 5*5 mean filter is selected.

[0251] Perception constraint: L visual =L PCP (enhaced,GT)=(VGG(enhanced),VGG(GT)) 2 , β represents the constraint strength of the perception constraint item, which can usually be a fixed value and can be adjusted according to the actual application scenario. VGG uses the publicly available VGG19 network to extract high-dimensional features.

[0252] Ignore the offset constraint: L shift =|enhanced,Alignment(GT)|1, ρ represents the constraint strength of the constraint item ignoring the offset, which can usually be a fixed value and can be adjusted according to the actual application scenario. Alignment is an alignment function, and the specific alignment method is not limited. In this embodiment, the initial alignment can be achieved by selecting the closest pixel value within a certain range.

[0253] Therefore, in an embodiment of the present application, when updating the first neural network, constraints such as features, colors, perception or ignored offsets are added to the loss function as constraints for updating the first neural network, so that the training of the first neural network can be achieved using non-aligned samples, so that the color, contrast, dynamic range and other output images of the first neural network are better, thereby improving the user experience.

[0254] In a possible implementation, the loss function may include a constraint term of a high-dimensional feature domain, which may be obtained by a fully convolutional neural network VGG19. By maximizing the similarity between the output image features and the true value image features in the feature domain, it is further ensured that the output image is clear and real.

[0255] Exemplarily, the structure of the first neural network can be as follows Fig.19 As shown, the first neural network can downsample the input image, thereby processing images of different scales obtained after downsampling once or multiple times. Three scales are used as an example for illustrative explanation here. In actual application scenarios, it can include the processing process of images of more scales or fewer scales.

[0256] like Fig.19 As shown in , each iteration process can be understood as the process of processing images of different scales. Fig.19 The first neural network module shown in FIG. 1 can be used to perform the aforementioned Figure 7 Detailed steps of the method for image processing shown in , as follows: the sampling module 19011 can be used to perform step 7011, the enhancement module 19013 can be used to perform step 7013, the global feature extraction module 19014 can be used to perform step 7014, the global information fusion module 19015 can be used to perform step 7015, and the up-sampling module 19016 can be used to perform step 7016. The sampling module 19021 can be used to perform step 7021, the scale fusion module 19022 can be used to perform step 7022, the enhancement module 19023 can be used to perform step 7023, the global feature extraction module 19024 can be used to perform step 7024, the global information fusion module 19025 can be used to perform step 7025, and the up-sampling module 19026 can be used to perform step 7026. The scale fusion module 19032 can be used to execute step 7032, the enhancement module 19033 can be used to execute step 7033, the global feature extraction module 19034 can be used to execute step 7034, and the global information fusion module 19035 can be used to execute step 7035.

[0257] The above-mentioned process of the method provided by the present application is described in detail. The structure of the device provided by the present application is described below. The present application provides an image processing device for performing the above-mentioned Figure 5-17D Corresponding steps, a training device is also provided for performing the above Figure 18-19 The corresponding steps.

[0258] First, see Fig. 20 , a schematic diagram of the structure of an image processing device provided in the present application.

[0259] Input module 2001, used for acquiring input images;

[0260] A local feature extraction module 2002, configured to obtain a first feature according to an input image, wherein the first feature is obtained according to at least one local feature extracted from the input image;

[0261] Enhancement module 2003, used to flip the values ​​of some features in the first feature to obtain a second feature, wherein the values ​​of some features before flipping are negatively correlated with the values ​​after flipping, and the some features are features that account for a preset proportion of the first feature;

[0262] The global feature extraction module 2004 is also used to obtain the global features corresponding to the input image;

[0263] A fusion module 2005 is used to fuse the second feature and the global feature to obtain an output feature;

[0264] The output module 2006 is used to obtain an output image according to the output features.

[0265] In a possible implementation, the enhancement module 2003 is specifically used to: split the first feature according to a preset quantity ratio to obtain a third feature and a fourth feature; flip the value of the fourth feature to obtain a fifth feature; and fuse the fifth feature and the third feature to obtain the second feature.

[0266] In a possible implementation, the enhancement module 2003 is specifically used to: perform feature extraction on the fifth feature and the third feature respectively to obtain an enhanced fifth feature and an enhanced third feature; and fuse the enhanced fifth feature and the enhanced third feature to obtain a second feature.

[0267] In a possible implementation, the enhancement module 2003 is specifically configured to invert the value of the fourth feature, and add a bias value to the inverted value to obtain the fifth feature.

[0268] In a possible implementation, the output feature is obtained by taking the input image as the input of the first neural network and outputting it by the first neural network, the first neural network is trained using a training set, the training set includes at least one non-aligned sample, an non-aligned sample includes an input image and a true value image, and the position of the target object included in the input image and the position of the target object included in the true value image are not exactly the same.

[0269] In a possible implementation, the image processing device further includes: a downsampling module 2007, which is used to downsample the input image before the local feature extraction module obtains the first feature based on the input image to obtain a downsampled input image, and the updated input image is used to obtain the first feature.

[0270] In a possible implementation, the image processing device further includes: an upsampling module 2008, which is used to upsample the output features to obtain upsampled features before the output module obtains the output image based on the output features; a local feature extraction module 2002, which is specifically used to fuse the first feature and the upsampled feature to obtain an updated first feature, the updated first feature is used to update the second feature, the updated second feature is used to update the output feature, and the updated output feature is obtained.

[0271] See also Fig.21 , a schematic structural diagram of a training device provided in the present application.

[0272] The training device comprises:

[0273] An input module 2101, used to use samples in the training set as input of the first neural network;

[0274] The training module 2102 is used to iteratively train the first neural network to obtain a trained first neural network, wherein the training set includes one or more non-aligned samples, the non-aligned samples include an input image and a true value image, and the position of the target object included in the input image is not completely the same as the position of the target object included in the true value image; wherein, the process of any iterative training includes taking the one or more non-aligned samples as the input of the first neural network to obtain an output image, then calculating the difference between the output image and the true value image corresponding to the input image through a loss function to obtain a loss value, and updating the first neural network according to the loss value to obtain the first neural network with the updated output of the current iteration.

[0275] In a possible implementation, the loss function may include a constraint term that ignores the offset between the output image and the true image, where the offset refers to the offset between the position of the same target object in the output image and the position in the true image. Therefore, in the implementation of the present application, even when training with non-aligned samples, clear and realistic output images can be obtained through training, and the effect is better, it can adapt to more scenes, and the generalization ability is stronger.

[0276] See also Fig. 22 , a structural schematic diagram of another image processing device provided by the present application is described as follows.

[0277] The training device may include a processor 2201 and a memory 2202. The processor 2201 and the memory 2202 are interconnected via a line. The memory 2202 stores program instructions and data.

[0278] The memory 2202 stores the aforementioned Figure 5-17D The program instructions and data corresponding to the steps in .

[0279] Processor 2201 is used to execute the above Figure 5-17D The method steps shown in any embodiment.

[0280] Optionally, the image processing apparatus may further include a transceiver 2203 for receiving or sending data.

[0281] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a program for generating a vehicle driving speed is stored. When the computer is running on a computer, the computer executes the above-mentioned Figure 7-Figure 9 The illustrated embodiments describe steps in a method.

[0282] Optionally, the aforementioned Fig. 22 The image processing device shown in is a chip.

[0283] See also Fig.23 , a structural schematic diagram of another training device provided in the present application is described as follows.

[0284] The training device may include a processor 2301 and a memory 2302. The processor 2301 and the memory 2302 are interconnected via a line. The memory 2302 stores program instructions and data.

[0285] The memory 2302 stores the aforementioned Figure 18-19 The program instructions and data corresponding to the steps in .

[0286] Processor 2301 is used to execute the above Figure 18-19 The method steps are performed by the training device shown in any embodiment.

[0287] Optionally, the training device may further include a transceiver 2303 for receiving or sending data.

[0288] In an embodiment of the present application, a computer-readable storage medium is also provided, in which a program for generating a vehicle driving speed is stored. When the computer is running on a computer, the computer executes the above-mentioned Figure 7-Figure 9 The illustrated embodiments describe steps in a method.

[0289] Optionally, the aforementioned Fig.23 The training device shown in is a chip.

[0290] The present application also provides an image processing device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 5-17DThe method steps shown in any embodiment.

[0291] The present application also provides a training device, which can also be called a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is used to execute the aforementioned Figure 18-19 The steps of the method shown in any embodiment.

[0292] The embodiment of the present application also provides a digital processing chip. The digital processing chip integrates a circuit and one or more interfaces for implementing the functions of the above-mentioned processor 2201, processor 2301, or processor 2201, processor 2301. When the digital processing chip integrates a memory, the digital processing chip can complete the method steps of any one or more of the above-mentioned embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory through a communication interface. The digital processing chip implements the method steps in the above-mentioned embodiment according to the program code stored in the external memory.

[0293] The present application also provides a computer program product which, when executed on a computer, enables the computer to execute the above Figure 5-17D Any embodiment or Figure 18-19 The steps in the method described in any embodiment.

[0294] The training device provided in the embodiment of the present application may be a chip, and the chip includes: a processing unit and a communication unit, the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit. The processing unit may execute the computer execution instructions stored in the storage unit to enable the chip in the server to execute the above Figure 7-Figure 9 The deep learning training method for a computing device described in the illustrated embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0295] Specifically, the aforementioned processing unit or processor may be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.

[0296] For example, see Fig.24 , Fig.24 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 240, NPU 240 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2403, which is controlled by the controller 2404 to extract matrix data from the memory and perform multiplication operations.

[0297] In some implementations, the operation circuit 2403 includes multiple processing units (process engines, PEs) inside. In some implementations, the operation circuit 2403 is a two-dimensional systolic array. The operation circuit 2403 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2403 is a general-purpose matrix processor.

[0298] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2402 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2401 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 2408.

[0299] The unified memory 2406 is used to store input data and output data. The weight data is directly transferred to the weight memory 2402 through the direct memory access controller (DMAC) 2405. The input data is also transferred to the unified memory 2406 through the DMAC.

[0300] The bus interface unit (BIU) 2410 is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2409 .

[0301] The bus interface unit 2410 (BIU) is used for the instruction fetch memory 2409 to obtain instructions from the external memory, and is also used for the storage unit access controller 2405 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0302] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2406 or to transfer weight data to the weight memory 2402 or to transfer input data to the input memory 2401.

[0303] The vector calculation unit 2407 includes multiple operation processing units, and further processes the output of the operation circuit when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as batch normalization, pixel-level summation, upsampling of feature planes, etc.

[0304] In some implementations, the vector calculation unit 2407 can store the processed output vector to the unified memory 2406. For example, the vector calculation unit 2407 can apply a linear function and / or a nonlinear function to the output of the operation circuit 2403, such as linear interpolation of the feature plane extracted by the convolution layer, and then, for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2407 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2403, for example, for use in a subsequent layer in a neural network.

[0305] An instruction fetch buffer 2409 connected to the controller 2404 is used to store instructions used by the controller 2404;

[0306] Unified memory 2406, input memory 2401, weight memory 2402 and instruction fetch memory 2409 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0307] Among them, the operations of each layer in the recurrent neural network can be performed by the operation circuit 2403 or the vector calculation unit 2407.

[0308] The processor mentioned in any of the above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors for controlling the above Figure 5-17D Any embodiment or Figure 18-19 An integrated circuit is implemented by a program of the method of any one of the embodiments.

[0309] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0310] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0311] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0312] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state drive (SSD)), etc.

[0313] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0314] Finally, it should be noted that the above are only specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A method for image processing, characterized in that: include: Get the input image; Acquire a first feature according to the input image, where the first feature is obtained according to at least one local feature extracted from the input image; Flipping the values ​​of some features in the first features to obtain a second feature, wherein the values ​​of the some features before flipping are negatively correlated with the values ​​after flipping, and the some features are features with a preset quantity ratio in the first features; Acquire global features of the input image; fusing the second feature and the global feature to obtain an output feature; The output image is obtained according to the output feature; wherein the value of some features in the first feature is flipped to obtain the second feature, including: Splitting the first feature according to the preset quantity ratio to obtain a third feature and a fourth feature; Flipping the value of the fourth feature to obtain a fifth feature; The fifth feature and the third feature are combined to obtain the second feature.

2. The method according to claim 1, characterized in that: The fusing the fifth feature and the third feature to obtain the second feature includes: Performing feature extraction on the fifth feature and the third feature respectively to obtain an enhanced fifth feature and an enhanced third feature; The second feature is obtained by fusing the enhanced fifth feature and the enhanced third feature.

3. The method according to claim 1 or 2, characterized in that: The step of flipping the value of the fourth feature to obtain the fifth feature includes: The value of the fourth feature is inverted, and a bias value is added to the inverted value to obtain the fifth feature.

4. The method according to claim 1 or 2, characterized in that: The output feature is obtained by taking the input image as the input of a first neural network and outputting the first neural network, wherein the first neural network is trained using a training set, wherein the training set includes at least one non-aligned sample, and one of the non-aligned samples includes the input image and a true value image, and a position of a target object included in the input image and a position of the target object included in the true value image are not exactly the same.

5. The method according to claim 1 or 2, characterized in that: Before acquiring the first feature according to the input image, the method further includes: Down-sample the input image to obtain an updated input image.

6. The method according to claim 1 or 2, characterized in that: Before obtaining the output image according to the output feature, the method further includes: Upsampling the output features to obtain upsampled features; The obtaining a first feature according to the input image comprises: The first feature and the up-sampled feature are fused to obtain an updated first feature, the updated first feature is used to update the second feature, and the updated second feature is used to update the output feature to obtain the updated output feature.

7. An image processing device, characterized in that: include: An input module, used to obtain an input image; A local feature extraction module, used to obtain a first feature according to the input image, where the first feature is obtained according to at least one local feature extracted from the input image; an enhancement module, configured to flip the values ​​of some features in the first features to obtain a second feature, wherein the values ​​of the some features before flipping are negatively correlated with the values ​​after flipping, and the some features are features of a preset quantity ratio in the first features; The global feature extraction module is also used to obtain the global features corresponding to the input image; A fusion module, used for fusing the second feature and the global feature to obtain an output feature; An output module, used for obtaining an output image according to the output feature; Wherein, the enhancement module is specifically used for: Splitting the first feature according to the preset quantity ratio to obtain a third feature and a fourth feature; Flipping the value of the fourth feature to obtain a fifth feature; The fifth feature and the third feature are combined to obtain the second feature.

8. The device according to claim 7, characterized in that The enhancement module is specifically used for: Performing feature extraction on the fifth feature and the third feature respectively to obtain an enhanced fifth feature and an enhanced third feature; The second feature is obtained by fusing the enhanced fifth feature and the enhanced third feature.

9. The device according to claim 7 or 8, characterized in that The enhancement module is specifically used to invert the value of the fourth feature and add a bias value to the inverted value to obtain the fifth feature.

10. The device according to claim 7 or 8, characterized in that The output feature is obtained by taking the input image as the input of a first neural network and outputting the first neural network, wherein the first neural network is trained using a training set, wherein the training set includes at least one non-aligned sample, and one of the non-aligned samples includes the input image and a true value image, and a position of a target object included in the input image and a position of the target object included in the true value image are not exactly the same.

11. The device according to claim 7 or 8, characterized in that The image processing device further includes: A downsampling module is used to downsample the input image before the local feature extraction module obtains the first feature based on the input image, so as to obtain the updated input image, and the updated input image is used to obtain the first feature.

12. The device according to claim 7 or 8, characterized in that The image processing device further includes: An upsampling module, used for upsampling the output features to obtain upsampled features before the output module obtains an output image according to the output features; The local feature extraction module is also used to fuse the first feature and the up-sampled feature to obtain an updated first feature, the updated first feature is used to update the second feature, and the updated second feature is used to update the output feature to obtain the updated output feature.

13. An image processing device, characterized in that: The method comprises a processor, wherein the processor is coupled to a memory, wherein the memory stores a program, and when the program instructions stored in the memory are executed by the processor, the method according to any one of claims 1 to 6 is implemented.

14. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method according to any one of claims 1 to 6.

Images