Image processing method, device, equipment and storage medium
By introducing candidate operation methods into convolutional neural networks and determining the optimal operation methods using weighted probability, the first convolutional neural subnet was constructed, which solved the problem of poor image reconstruction quality in the prior art and realized automated image reconstruction The process improves image quality and reduces network computing power.
Patent Information
- Application Number
- CN202010774449.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2020-08-04
- Publication Date
- 2025-02-14
- Estimated Expiration
- 2040-08-04
AI Technical Summary
The existing hyper-analysis method based on deep neural networks requires manual participation in the design of network architecture and performing a large number of architectural hyperparameter adjustments, resulting in poor image reconstruction quality.
By introducing candidate operation methods of input layer and intermediate layer into the convolutional neural network, and determining the optimal operation method using weighted probability, the first convolutional neural subnet is constructed, thereby improving the image reconstruction quality.
This method can automatically determine the optimal operation mode between nodes, improve the reconstruction quality of images, reduce network computing power, and reduce manual intervention.
Smart Images
Figure CN114066726B_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image processing method, apparatus, device and storage medium. Background Art
[0002] With the rapid development of video and image super-resolution technology, deep learning-based methods have shown great advantages over traditional methods and have quickly become a research hotspot in the field of video and image super-resolution technology.
[0003] Among the existing super-resolution methods based on deep neural networks, whether they are the Generative Adversarial Networks (GAN)-based methods that focus on generating more realistic visual effects, or the fully convolutional neural network-based methods that focus on higher objective indicators, both require manual participation in the design of specific network architectures, and the network testing and training process may also be accompanied by a large amount of architectural hyperparameter adjustment work, and ultimately only suboptimal video and image reconstruction effects can be achieved.
[0004] Application Contents
[0005] The embodiments of the present application provide an image processing method, apparatus, device and storage medium to improve the reconstruction quality of an image.
[0006] In a first aspect, an embodiment of the present application provides an image processing method, comprising:
[0007] Get the image to be processed;
[0008] Inputting the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the resolution of the target image is greater than the resolution of the image to be processed;
[0009] Among them, the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node contained in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer.
[0010] In a second aspect, an embodiment of the present application provides an image processing device, including:
[0011] An image acquisition module, used for acquiring an image to be processed;
[0012] A target image determination module, used for inputting the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the resolution of the target image is greater than the resolution of the image to be processed;
[0013] Among them, the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node contained in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer.
[0014] In a third aspect, an embodiment of the present application provides a computer device, including:
[0015] processor;
[0016] A memory for storing one or more programs;
[0017] When the one or more programs are executed by the processor, the processor implements the image processing method as described in the first aspect.
[0018] In a fourth aspect, an embodiment of the present application provides a storage medium on which a computer program is stored, characterized in that when the program is executed by a processor, the image processing method described in the first aspect is implemented.
[0019] The embodiments of the present application provide an image processing method, apparatus, device and storage medium, wherein an acquired image to be processed is input into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node contained in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer. This scheme uses the operation mode with the largest weighted probability among the candidate operation modes of the input layer as the operation mode of the input node on the input information, and uses the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer as the operation mode of the intermediate node on the input information, to obtain a convolutional neural network, and when the convolutional neural network is used to reconstruct a low-resolution image to be processed into a high-resolution target image, the reconstruction quality of the image is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application;
[0021] Figure 2A schematic diagram of a first convolutional neural sub-network provided in an embodiment of the present application;
[0022] Figure 3 A flowchart of another image processing method provided in an embodiment of the present application;
[0023] Figure 4 A schematic diagram of an initial first convolutional neural sub-network provided in an embodiment of the present application;
[0024] Figure 5 A schematic diagram of a first convolutional neural sub-network determined by an inter-node operation mode provided in an embodiment of the present application;
[0025] Figure 6 A schematic diagram of a convolutional neural network provided in an embodiment of the present application;
[0026] Figure 7 A structural diagram of an image processing device provided in an embodiment of the present application;
[0027] Figure 8 A structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0028] The present application will be further described in detail below in conjunction with the accompanying drawings and embodiments. It will be appreciated that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only the parts related to the present application, rather than all structures, are shown in the accompanying drawings. In addition, the embodiments in the present application and the features in the embodiments may be combined with each other without conflict.
[0029] Figure 1 This is a flowchart of an image processing method provided in an embodiment of the present application. This embodiment is applicable to the case of image super-resolution reconstruction, for example, a low-resolution image can be reconstructed into a high-resolution image. When reconstructing a low-resolution image into a high-resolution image, multiple low-resolution images can be synthesized into a high-resolution image, or a high-resolution image can be obtained based on a single low-resolution image. This embodiment takes the latter as an example. The method can be executed by an image processing device, which can be implemented in software and / or hardware, and can be integrated in a computer device such as a notebook, a cloud server or a PDA. Reference Figure 1 , the method may include the following steps:
[0030] S110: Obtain an image to be processed.
[0031] The image to be processed may be a low-resolution picture or video, so as to convert the low-resolution picture or video into a high-resolution picture or video. The low-resolution picture or video may be an original picture or video, or a pre-processed picture or video. In this embodiment, the pre-processed picture or video is used as an example of the image to be processed, which can improve the reconstruction effect of the image. The pre-processing may include but is not limited to operations such as filtering and noise reduction.
[0032] S120. Input the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the resolution of the target image is greater than the resolution of the image to be processed.
[0033] Among them, the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node contained in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer.
[0034] The convolutional neural network of this embodiment is used to convert a low-resolution image into a high-resolution image, and therefore can also be referred to as a super-resolution network or super-resolution neural network. The first convolutional neural network is a component of the convolutional neural network, and can also be referred to as a cell or block, and is mainly used to extract image features of the image to be processed. Figure 2 , Figure 2 A schematic diagram of a first convolutional neural sub-network provided in an embodiment of the present application. The first convolutional neural sub-network includes an input layer, an intermediate layer, and an output layer. The number of layers of the input layer, the intermediate layer, and the output layer, as well as the number of nodes contained in each layer, can be set according to actual needs. For example Figure 2In the first convolutional neural network shown, the input layer includes two layers, the intermediate layer includes four layers, and the output layer includes one layer. Each layer includes a node, and different nodes are located in different layers. For example, node0 and node1 are located in the first layer and the second layer respectively, as input nodes, node2-node 5 are located in the third layer to the sixth layer respectively, as intermediate nodes, and node 6 is located in the seventh layer as an output node. Each node represents a data state. Two associated nodes can convert image data from one state to another state by performing a certain operation. For example, the image data on node0 can be output to node2 after performing a skip connection operation, and the image data on node0 can be output to node5 after performing a 3x3 depth separation convolution operation. The operation modes between different nodes can be the same or different. In practical applications, the convolutional neural network can include multiple first convolutional neural subnetworks. When multiple first convolutional neural subnetworks are included, these multiple first convolutional neural subnetworks can be repeatedly stacked. Wherein node0 represents the output of the previous first convolutional neural subnetwork, node1 represents the output of the previous first convolutional neural subnetwork, and the data of node0 and node1 in the initial first convolutional neural subnetwork are the same, both of which are image data of the image to be processed. In this embodiment, the image data can be represented in the form of a tensor (b, d, h, w), where b represents the number of images to be processed, d represents the depth, h represents the height of the image to be processed, and w represents the width of the image to be processed.
[0035] Taking the example of a convolutional neural network including multiple first convolutional neural subnetworks, the input information of the current first convolutional neural subnetwork is the output of the previous first convolutional neural subnetwork and the previous previous first convolutional neural subnetwork. The input of the intermediate node can be one or more. When there are multiple, the input information can come from the input node, for example Figure 2The input information of node5 in the example comes from input nodes node0 and node1, or partly from input nodes and partly from other intermediate nodes. For example, the input information of intermediate node node3 comes from input node node1 and intermediate node node2 respectively. Taking the intermediate node of this embodiment as an example including two inputs, it can not only reduce the complexity of the structure of the first convolutional neural sub-network and the network computing power, but also improve the reconstruction effect of the image. The first operation is the operation performed by the input node on the input information, and the second operation is the operation performed by the intermediate node on the input information. The candidate operation mode is a pre-set operation mode for changing the data state. Optionally, the candidate operation mode may include but is not limited to no connection, 3x3 average pooling operation, 3x3 maximum pooling operation, skip connection operation, 3x3 depth separation convolution operation, 5x5 depth separation convolution operation, 3x3 atrous convolution operation, hole size is 2 and 5x5 atrous convolution operation, hole size is 2. The input layer candidate operation mode is the candidate operation mode corresponding to the input node and the intermediate node, and the intermediate layer candidate operation mode is the candidate operation mode corresponding to the intermediate nodes. The input layer candidate operation mode and the intermediate layer candidate operation mode can be the same or different. This embodiment takes the same as an example.
[0036] Each operation mode corresponds to a weighted probability, which can be represented by the architecture hyperparameter corresponding to the operation mode and conforms to the softmax function form. The weighted probability corresponding to the candidate operation mode of the input layer may be different from the weighted probability corresponding to the candidate operation mode of the intermediate layer. In this embodiment, the input node operates on the input information in the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the intermediate node operates on the input information in the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer. This can ensure that the structure of the first convolutional neural subnetwork is optimal, and thus ensure the optimal quality of the reconstructed image. It should be noted that the nodes in the intermediate layer used to perform the second operation on the input information may be some of the intermediate nodes or all of the intermediate nodes. For example Figure 2 The intermediate nodes node4 and node5 do not perform the second operation on the input information, while the intermediate nodes node2 and node3 perform the second operation on the input information.
[0037] The output node of the first convolutional neural network is used to combine the output results of the intermediate nodes and pass the combined results to other sub-networks of the convolutional neural network, such as the upsampling sub-network, through which the image reconstruction is completed. The combination method can be selected according to the actual situation. For example, the output results of the intermediate nodes can be calculated and output by element-by-element summation, or they can be spliced and output along a certain dimension, so as to combine the four image data into one image data. Figure 2As shown, when an intermediate node has multiple inputs, the intermediate node can also combine the multiple inputs to obtain a type of image data, and then output it to the output node. The combination method is similar to that of the output node.
[0038] Specifically, the image to be processed can be input into the first convolutional neural sub-network, so that the input node of the first convolutional neural sub-network performs a first operation on the input information, obtains a first operation result, and sends it to the intermediate node of the first convolutional neural sub-network; performs a second operation on the received input information through the intermediate node to obtain a second operation result, and sends the output result of the intermediate node to the output node; and performs upsampling processing on the result output by the output node to obtain the target image.
[0039] The embodiment of the present application provides an image processing method, which inputs an acquired image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, and the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the intermediate node contained in the intermediate layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer. The method uses the operation mode with the largest weighted probability among the candidate operation modes of the input layer as the operation mode of the input node on the input information, and uses the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer as the operation mode of the intermediate node on the input information to obtain a convolutional neural network. When the convolutional neural network is used to reconstruct a low-resolution image to be processed into a high-resolution target image, the reconstruction quality of the image is improved.
[0040] Before obtaining the image to be processed, it is necessary to determine the structure of the first convolutional neural sub-network, that is, to determine the operation mode between the associated nodes in the first convolutional neural sub-network. In one case, the operation mode between the associated nodes can be obtained by training the first convolutional neural network, thereby obtaining the structure of the first convolutional neural sub-network. For example, refer to Figure 3 , Figure 3 A flowchart of another image processing method provided in an embodiment of the present application.
[0041] S210: Input the training sample into the initial convolutional neural network, so that the input node of the first convolutional neural sub-network performs a first mixed weighted operation on the received input information.
[0042] Among them, the operation mode of the first hybrid weighted operation is determined according to the candidate operation mode of the input layer and the corresponding weighted probability. Optionally, the low-resolution original image can be randomly cropped to form a patch, and then randomly performed data enhancement processing such as horizontal flipping, vertical flipping, 90-degree rotation, etc. to obtain a training sample to train the first convolutional neural sub-network. In one case, the Neural Architecture Search (NAS) technology can be combined with the convolutional neural network to automatically search for the optimal operation mode between nodes to ensure that the structure of the first convolutional neural sub-network is optimal, thereby improving the image reconstruction effect, for example, the similarity between the target image and the real high-resolution image can be improved, and the peak signal-to-noise ratio can be improved. Optionally, the optimal operation mode can be searched in a hybrid weighted manner, and the hybrid weighting is to combine the architecture hyperparameters with the candidate operation mode in the form of weighted probability. Hybrid weighting means that there is no longer a single operation mode between any nodes in the first convolutional neural sub-network, but a hybrid weighting of all candidate operation modes. The training of the first convolutional neural sub-network depends on the complete convolutional neural network. In this embodiment, the convolutional neural network in which the operation mode between nodes is a hybrid weighted operation mode is called the initial convolutional neural network. Exemplarily, the mixed weighted operation result can be obtained by using the following formula:
[0043]
[0044] in, represents the mixed weighted operation result, o(x) represents a single operation mode between associated nodes, O is the candidate operation set, and a k is the architecture hyperparameter corresponding to the k-th operation mode, is the weighted coefficient corresponding to the operation modes between different associated nodes, which conforms to the form of softmax function, represents the weighted probability of the kth operation mode, and takes the value [0,1]. In this embodiment, eight candidate operation modes are included as an example, and eight weighted probabilities are corresponding. Accordingly, eight candidate operation modes correspond to any associated nodes, and the sum of the eight weighted probabilities corresponding to the eight candidate operation modes between the same group of associated nodes is 1. For example, refer to Figure 4 , Figure 4 A schematic diagram of an initial first convolutional neural sub-network provided in an embodiment of the present application is provided for ease of viewing. Figure 4 Take three candidate operation modes between the associated nodes as an example. The three arrows between the same group of associated nodes represent three candidate operation modes. The operation mode of the first mixed weighted operation is the operation mode obtained by mixing and weighting all the candidate operation modes between the input node and the intermediate node. The input node performs the first mixed weighted operation on the input information, obtains the first mixed weighted result, and transmits it to the intermediate node.
[0045] S220, sending the first mixed weighted operation result output by the input node to the intermediate node, so that the intermediate node performs a second mixed weighted operation on the received input information, and sending the output result of the intermediate node to the output node of the first convolutional neural sub-network.
[0046] The operation mode of the second hybrid weighted operation is determined according to the candidate operation modes of the intermediate nodes and the corresponding weighted probabilities. The operation mode of the second hybrid weighted operation is the operation mode obtained by hybrid weighting all the candidate operation modes between the associated intermediate nodes, and the process is similar to the first hybrid weighted operation. Figure 4 As shown, taking the intermediate nodes node2 and node3 as examples, the intermediate node node2 combines the received input information, then performs a second mixed weighted operation, and sends the second mixed operation result to the intermediate node node3. The weighted probabilities corresponding to the candidate operation modes in the operation mode of the second mixed weighted operation may be different from the weighted probabilities corresponding to the candidate operation modes in the operation mode of the first mixed weighted operation. During the training process, the weighted probabilities corresponding to the candidate operation modes in the operation mode of the second mixed weighted operation may be changed from the weighted probabilities corresponding to the candidate operation modes in the operation mode of the first mixed weighted operation. The output result of the intermediate node is the result obtained by combining the input information received by the intermediate node, for example Figure 2 The intermediate node node3 combines the output results of the intermediate node node2 and the input node node1 and sends them to the output node node6. The intermediate node node5 combines the output results of the input node node0 and the input node node1 and sends them to the output node node6. The output node node6 combines the output results of the intermediate nodes node2, node3, node4 and node5 to obtain the output result and sends it to the subsequent part of the initial convolutional neural network.
[0047] S230 , performing up-sampling processing on the result output by the output node to obtain an up-sampled image.
[0048] Upsampling is used to amplify the result output by the output node to obtain a high-resolution image, namely an upsampled image. The upsampled image is used to train the architecture hyperparameters of the first convolutional neural network. The output node combines the results sent by the intermediate nodes and sends them to the subsequent part of the initial convolutional neural network to achieve image amplification. Optionally, a 3x3 convolution operation can be performed on the result output by the output node, and the convolution result is called a feature map. Then, the feature map is amplified by pixelshuffle (2) to obtain a verification image. Among them, the 3x3 convolution operation is used to prevent the image quality from changing dramatically during the amplification process. Pixelshuffle (2) represents the transformation of the tensor [height, width, 4*depth] to [2*height, 2*width, depth], that is, a tensor shape transformation method that expands the spatial dimension by compressing the depth dimension while keeping the total amount of tensor data unchanged. The function of pixelshuffle can also be replaced by deconvolution. This embodiment does not limit the number of convolutions and amplifications. For example, the result output by the output node can be convolved and amplified once to obtain an upsampled image, or it can be convolved and amplified multiple times to obtain an upsampled image.
[0049] S240. When the upsampled image meets the preset conditions, the candidate operation mode with the largest weighted probability among the candidate operation modes of the input layer corresponding to the upsampled image that meets the preset conditions is used as the operation mode of the first operation, and the candidate operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer is used as the operation mode of the second operation, to obtain the first convolutional neural sub-network.
[0050] Training the first convolutional neural subnetwork, that is, determining the architecture hyperparameters corresponding to the mixed weighted operation mode in the first convolutional neural subnetwork, when the upsampled image meets the preset conditions, the training is considered to be completed. Since the initial convolutional neural network includes not only the architecture hyperparameters corresponding to the mixed weighted operation mode, but also the weight coefficients corresponding to the convolution operation, the weight parameters will also affect the reconstruction effect of the image, so the weight coefficients and architecture hyperparameters can be trained. Correspondingly, the preset conditions may include that the weight loss value of the upsampled image and the real image obtained by the training sample under the current weight parameters tends to the first convergence condition, and the corresponding architecture loss value under the current architecture hyperparameters tends to the second convergence condition. Among them, the first convergence condition can be that the decrease in the weight loss value gradually decreases and becomes stable, and no longer decreases; the second convergence condition can be that the decrease in the architecture loss value gradually decreases and becomes stable, and no longer decreases. Specifically, the training process of the first convolutional neural subnetwork may include:
[0051] The initial architecture hyperparameters of the initial convolutional neural network are fixed unchanged, and a weight loss value of the initial convolutional neural network is determined according to the verification image and the real image corresponding to the training sample;
[0052] Adjusting the weight parameters of the initial convolutional neural network according to the weight loss value;
[0053] The weight parameters of the last update are fixed unchanged, the architecture loss value of the initial convolutional neural network is determined according to the training sample, and the architecture hyperparameters of the initial convolutional neural network are adjusted according to the architecture loss value;
[0054] When the weight loss value of the initial convolutional neural network satisfies the first convergence condition and the architecture loss value satisfies the second convergence condition, it is determined that the upsampled image satisfies the preset condition.
[0055] In this embodiment, the initial architecture hyperparameters of the initial convolutional neural network are the initial architecture hyperparameters included in the mixed weighted operation mode in the first convolutional neural subnetwork. In this embodiment, when training the first convolutional neural subnetwork, the weight parameters and the architecture hyperparameters are updated alternately. In order to complete the training of the first convolutional neural subnetwork, this embodiment divides the training samples into a training set and a validation set. The training set is used to train the weight parameters during training, and the validation set is used to verify the training effect of the first convolutional neural subnetwork during training, and is also used to train the architecture hyperparameters.
[0056] Optionally, the initial architecture hyperparameters can be fixed unchanged, and a single batch of data of the training set is used to achieve mapping from low resolution to high resolution through the initial convolutional neural network forward propagation, and the weight loss value Loss1 of the output high-resolution image and the true high-resolution image is calculated, and then the weight loss value Loss1 is used to calculate the gradient of the weight parameter through the initial convolutional neural network back propagation, and then the weight parameter is updated once. After the weight parameter is updated, the weight parameter updated last time is fixed unchanged, and the architecture loss value Loss2 is calculated through the initial convolutional neural network forward propagation using a single batch of data of the validation set, and then the gradient of the architecture hyperparameter is calculated through the initial convolutional neural network back propagation to complete the update of the architecture hyperparameter. Then the above process is performed alternately until the weight loss value Loss1 meets the first convergence condition and the architecture loss value Loss2 meets the second convergence condition. After the training is completed, the operation mode corresponding to the maximum value of a set of weighted probabilities represented by the architecture hyperparameters in the mixed weighted operation corresponding to the weight loss value Loss1 and the architecture loss value Loss2 meeting the conditions can be determined as the operation mode between the corresponding nodes.
[0057] Exemplary, reference Figure 5 , Figure 5A schematic diagram of the first convolutional neural subnetwork determined by an inter-node operation mode provided in an embodiment of the present application. Among them, sep_conv_3x3 represents a 3x3 depth separation convolution operation, dil_conv_3x3 represents a 3x3 atrous convolution operation with a hole size of 2, skip_connect represents a skip connection operation, and avg_pool_3x3 represents a 3x3 average pooling operation. After the structure of the first convolutional neural subnetwork is determined, the first convolutional neural subnetwork can be used to reconstruct a complete convolutional neural network in the same way as the initial convolutional neural network, except that the structure of the first convolutional neural subnetwork in the currently constructed convolutional neural network is no longer a mixed weighted structure, but a fixed optimal structure. It should be noted that the number of first convolutional neural subnetworks included in the initial convolutional neural network may be different from the number of first convolutional neural networks in the subsequent convolutional neural network constructed based on the first convolutional neural network with a determined structure. Considering that during the search, the first convolutional neural subnetwork needs to calculate the mixed weighting, which consumes more memory, usually, the number of first convolutional neural subnetworks in the initial convolutional neural network is small. In order to further improve the image reconstruction effect, the number of first convolutional neural networks in the subsequent convolutional neural network constructed based on the first convolutional neural subnetwork with a determined structure can be more. The weight coefficients need to be trained from scratch for the reconstructed convolutional neural network. At this time, the first convolutional neural subnetwork with a determined structure is used. There are no architecture hyperparameters. The training process is similar to the traditional training process based on the full convolutional neural network, which will not be repeated here.
[0058] S250: Obtain an image to be processed.
[0059] S260: Input the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network.
[0060] Exemplary, reference Figure 6 , Figure 6 A schematic diagram of a convolutional neural network provided in an embodiment of the present application, wherein the trained convolutional neural network comprises n first convolutional neural sub-networks, where the size of n can be set according to actual needs. Figure 6 Take the case of two magnifications. After preprocessing, the image to be processed is input into the first convolutional neural sub-network. After multiple first convolutional neural sub-networks are repeatedly stacked, two convolution and magnification processes are performed, and then another convolution operation is performed to obtain the target image. The subsequent convolution and magnification are the upsampling described in the above embodiment.
[0061] The embodiment of the present application provides an image processing method. On the basis of the above embodiment, the neural architecture search method is first combined with the convolutional neural network, and the optimal operation mode between related nodes is searched by using a mixed weighted method to obtain the optimal first convolutional neural sub-network structure, and then the convolutional neural network is reconstructed based on the first convolutional neural sub-network with a determined structure. The weight parameters of the reconstructed convolutional neural network are determined through training, and then the trained convolutional neural network is used to realize super-resolution reconstruction of the image, which not only improves the effect of super-resolution reconstruction, but also effectively reduces the network computing power.
[0062] In addition to using the above-mentioned hybrid weighted method to search for the optimal architecture of the first convolutional neural sub-network, other methods can also be used, such as using an RNN (recurrent neural network) controller based on a reinforcement learning method to search for the architecture of the first convolutional neural sub-network.
[0063] Figure 7 This is a structural diagram of an image processing device provided in an embodiment of the present application. The device can execute the image processing method provided in the above embodiment. Figure 7 , the device may include:
[0064] An image acquisition module 31 is used to acquire an image to be processed;
[0065] A target image determination module 32, used for inputting the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the resolution of the target image is greater than the resolution of the image to be processed;
[0066] Among them, the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node contained in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer.
[0067] The embodiment of the present application provides an image processing device, which inputs an acquired image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the first convolutional neural subnetwork contained in the convolutional neural network includes an input layer and an intermediate layer, and the operation mode of the first operation of the input node contained in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the intermediate node contained in the intermediate layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer. The device uses the operation mode with the largest weighted probability among the candidate operation modes of the input layer as the operation mode of the input node on the input information, and uses the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer as the operation mode of the intermediate node on the input information to obtain a convolutional neural network, and when the convolutional neural network is used to reconstruct a low-resolution image to be processed into a high-resolution target image, the reconstruction quality of the image is improved.
[0068] Based on the above embodiment, the device further includes:
[0069] An input module, used for inputting a training sample into an initial convolutional neural network before acquiring an image to be processed, so that an input node of the first convolutional neural subnetwork performs a first mixed weighted operation on the received input information, wherein an operation mode of the first mixed weighted operation is determined according to the candidate operation mode of the input layer and a corresponding weighted probability;
[0070] A sending module, configured to send the first mixed weighted operation result output by the input node to the intermediate node, so that the intermediate node performs a second mixed weighted operation on the received input information, and send the output result of the intermediate node to the output node of the first convolutional neural sub-network, wherein the operation mode of the second mixed weighted operation is determined according to the candidate operation mode of the intermediate node and the corresponding weighted probability;
[0071] An upsampling module, used for performing upsampling processing on the result output by the output node to obtain an upsampled image;
[0072] An operation mode determination module is used to, when the upsampled image meets the preset conditions, use the candidate operation mode with the largest weighted probability among the candidate operation modes in the input layer corresponding to the upsampled image that meets the preset conditions as the operation mode of the first operation, and use the candidate operation mode with the largest weighted probability among the candidate operation modes in the intermediate layer as the operation mode of the second operation, to obtain the first convolutional neural sub-network.
[0073] Based on the above embodiment, the target image determination module 32 is specifically used for:
[0074] Inputting the image to be processed into the first convolutional neural sub-network, causing the input node of the first convolutional neural sub-network to perform a first operation on the input information, obtaining a first operation result, and sending the first operation result to the intermediate node of the first convolutional neural sub-network;
[0075] Performing a second operation on the received input information through the intermediate node to obtain a second operation result, and sending an output result of the intermediate node to the output node;
[0076] An upsampling process is performed on the result output by the output node to obtain the target image.
[0077] Based on the above embodiment, the weighted probability is determined according to the architecture hyperparameter corresponding to the corresponding operation mode;
[0078] The upsampled image meets a preset condition, including:
[0079] The initial architecture hyperparameters of the initial convolutional neural network are fixed unchanged, and a weight loss value of the initial convolutional neural network is determined according to the upsampled image and the real image corresponding to the training sample;
[0080] Adjusting the weight parameters of the initial convolutional neural network according to the weight loss value;
[0081] The weight parameters of the last update are fixed unchanged, the architecture loss value of the initial convolutional neural network is determined according to the training sample, and the architecture hyperparameters of the initial convolutional neural network are adjusted according to the architecture loss value;
[0082] When the weight loss value of the initial convolutional neural network satisfies the first convergence condition and the architecture loss value satisfies the second convergence condition, it is determined that the upsampled image satisfies the preset condition.
[0083] Based on the above embodiment, the intermediate node includes two inputs, and the input comes from at least one of the input node and the intermediate node.
[0084] The image processing device provided in the embodiment of the present application can execute the image processing method in the above embodiment, and has the corresponding functional modules and beneficial effects of the execution method.
[0085] Figure 8 A structural diagram of a computer device provided in an embodiment of the present application, referring to Figure 8 The computer device may include: a processor 41, a memory 42, an input device 43 and an output device 44. The number of the processor 41 in the computer device may be one or more. Figure 8 A processor 41 is taken as an example. The processor 41 and the memory 42, the input device 43 and the output device 44 can be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.
[0086] The memory 42 is a computer-readable storage medium that can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the image processing method in the embodiment of the present application. The processor 41 executes various functional applications and data processing of the computer device by running the software programs, instructions and modules stored in the memory 42, that is, implementing the image processing method in the above embodiment.
[0087] The memory 42 mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application required for a function; the data storage area can store data created according to the use of the terminal, etc. In addition, the memory 42 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 42 may further include a memory remotely arranged relative to the processor 41, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0088] The input device 43 may be used to receive input digital or character information, and generate key signal input related to user settings and function control of the computer device. The output device 44 may include display devices such as display screens, and audio devices such as speakers and buzzers.
[0089] The computer device provided in the embodiment of the present application and the image processing method provided in the above embodiment belong to the same concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effect of executing the image processing method.
[0090] An embodiment of the present application further provides a storage medium on which a computer program is stored. When the program is executed by a processor, the image processing method as described in the above embodiment of the present application is implemented.
[0091] Of course, the storage medium containing computer executable instructions provided in the embodiment of the present application, whose computer executable instructions are not limited to the operations in the image processing method described above, can also execute related operations in the image processing method provided in any embodiment of the present application, and have corresponding functions and beneficial effects.
[0092] Through the above description of the implementation methods, technicians in the relevant field can clearly understand that the present application can be implemented with the help of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present application can essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a robot, a personal computer, a server, or a network device, etc.) to perform the image processing method described in the above embodiment of the present application.
Claims
1. An image processing method, characterized in that: include: Get the image to be processed; Inputting the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the resolution of the target image is greater than the resolution of the image to be processed; The first convolutional neural network included in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node included in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node included in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer; The convolutional neural network includes n first convolutional neural subnetworks, a first convolutional module, a second convolutional module, a third convolutional module, a first amplification module and a second amplification module, where n is an integer greater than 0; an input node of the input layer of each of the first convolutional neural subnetworks is the output of the previous first convolutional neural subnetwork of the first convolutional neural subnetwork, another input node of the input layer of the first convolutional neural subnetwork is the output of the previous first convolutional neural subnetwork of the first convolutional neural subnetwork, and the input nodes of the input layer of the initial first convolutional neural subnetwork are all the images to be processed; the last first convolutional neural subnetwork, the first convolutional module, the first amplification module, the second convolutional module, the second amplification module and the third convolutional module of the n first convolutional neural subnetworks are connected in sequence; Before obtaining the image to be processed, it also includes: Inputting the training sample into the initial convolutional neural network, so that the input node of the first convolutional neural sub-network performs a first mixed weighted operation on the received input information, wherein the operation mode of the first mixed weighted operation is determined according to the candidate operation mode of the input layer and the corresponding weighted probability; Sending the first mixed weighted operation result output by the input node to the intermediate node, so that the intermediate node performs a second mixed weighted operation on the received input information, and sending the output result of the intermediate node to the output node of the first convolutional neural sub-network, wherein the operation mode of the second mixed weighted operation is determined according to the candidate operation mode of the intermediate node and the corresponding weighted probability; Performing up-sampling processing on the result output by the output node to obtain an up-sampled image; When the upsampled image meets the preset condition, the candidate operation mode with the largest weighted probability among the candidate operation modes of the input layer corresponding to the upsampled image that meets the preset condition is used as the operation mode of the first operation, and the candidate operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer is used as the operation mode of the second operation, to obtain the first convolutional neural sub-network; The weighted probability is determined according to the architecture hyperparameter corresponding to the corresponding operation mode; the upsampled image meets the preset conditions, including: The initial architecture hyperparameters of the initial convolutional neural network are fixed unchanged, and a weight loss value of the initial convolutional neural network is determined according to the upsampled image and the real image corresponding to the training sample; Adjusting the weight parameters of the initial convolutional neural network according to the weight loss value; The weight parameters of the last update are fixed unchanged, the architecture loss value of the initial convolutional neural network is determined according to the training sample, and the architecture hyperparameters of the initial convolutional neural network are adjusted according to the architecture loss value; When the weight loss value of the initial convolutional neural network satisfies the first convergence condition and the architecture loss value satisfies the second convergence condition, it is determined that the upsampled image satisfies the preset condition.
2. The method according to claim 1, characterized in that The intermediate node includes two inputs; The two inputs come from different input nodes; or, one of the two inputs comes from an input node, and the other input comes from another intermediate node.
3. An image processing device, characterized in that: include: An image acquisition module, used for acquiring an image to be processed; A target image determination module, used for inputting the image to be processed into a convolutional neural network to obtain a target image output by the convolutional neural network, wherein the resolution of the target image is greater than the resolution of the image to be processed; The first convolutional neural network included in the convolutional neural network includes an input layer and an intermediate layer, the operation mode of the first operation of the input node included in the input layer on the input information is the operation mode with the largest weighted probability among the candidate operation modes of the input layer, and the operation mode of the second operation of the input information by the intermediate node included in the intermediate layer is the operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer; The convolutional neural network includes n first convolutional neural subnetworks, a first convolutional module, a second convolutional module, a third convolutional module, a first amplification module and a second amplification module, where n is an integer greater than 0; an input node of the input layer of each of the first convolutional neural subnetworks is the output of the previous first convolutional neural subnetwork of the first convolutional neural subnetwork, another input node of the input layer of the first convolutional neural subnetwork is the output of the previous first convolutional neural subnetwork of the first convolutional neural subnetwork, and the input nodes of the input layer of the initial first convolutional neural subnetwork are all the images to be processed; the last first convolutional neural subnetwork, the first convolutional module, the first amplification module, the second convolutional module, the second amplification module and the third convolutional module of the n first convolutional neural subnetworks are connected in sequence; The device also includes: An input module, used for inputting a training sample into an initial convolutional neural network before acquiring an image to be processed, so that an input node of the first convolutional neural subnetwork performs a first mixed weighted operation on the received input information, wherein an operation mode of the first mixed weighted operation is determined according to the candidate operation mode of the input layer and a corresponding weighted probability; A sending module, configured to send the first mixed weighted operation result output by the input node to the intermediate node, so that the intermediate node performs a second mixed weighted operation on the received input information, and send the output result of the intermediate node to the output node of the first convolutional neural sub-network, wherein the operation mode of the second mixed weighted operation is determined according to the candidate operation mode of the intermediate node and the corresponding weighted probability; An upsampling module, used for performing upsampling processing on the result output by the output node to obtain an upsampled image; an operation mode determination module, configured to, when the upsampled image satisfies a preset condition, use the candidate operation mode with the largest weighted probability among the candidate operation modes of the input layer corresponding to the upsampled image that satisfies the preset condition as the operation mode of the first operation, and use the candidate operation mode with the largest weighted probability among the candidate operation modes of the intermediate layer as the operation mode of the second operation, to obtain the first convolutional neural sub-network; The weighted probability is determined according to the architecture hyperparameter corresponding to the corresponding operation mode; the upsampled image meets the preset conditions, including: The initial architecture hyperparameters of the initial convolutional neural network are fixed unchanged, and a weight loss value of the initial convolutional neural network is determined according to the upsampled image and the real image corresponding to the training sample; Adjusting the weight parameters of the initial convolutional neural network according to the weight loss value; The weight parameters of the last update are fixed unchanged, the architecture loss value of the initial convolutional neural network is determined according to the training sample, and the architecture hyperparameters of the initial convolutional neural network are adjusted according to the architecture loss value; When the weight loss value of the initial convolutional neural network satisfies the first convergence condition and the architecture loss value satisfies the second convergence condition, it is determined that the upsampled image satisfies the preset condition.
4. A computer device, characterized in that: include: processor; A memory for storing one or more programs; When the one or more programs are executed by the processor, the processor implements the image processing method according to claim 1 or 2.
5. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image processing method according to claim 1 or 2 is implemented.