A method, chip, device and electronic device for image processing

Abstract

The embodiment of the present application discloses a method, chip, device and electronic device for image processing, which uses an image processing neural network to reconstruct the original image to obtain a super-resolution image; wherein the image processing neural network is a UNET network searched based on a differentiable neural network structure; wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module. In this way, the UNET network can be used to combine deep-level features with shallow-level features, refine the image, and improve the reconstruction effect of the super-resolution image. Each layer of the channel attention module performs channel learning on the feature map data from the two layers of downsampling modules in the encoder, enhances the attention to channel features, realizes selective reuse of channels, and further improves the performance of the image processing neural network and the reliability of the image super-resolution results.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a method, chip, device and electronic device for image processing. Background Art

[0002] With the widespread adoption of high-definition devices, the resolution of user-side display devices has generally increased to 2K or even higher. Image super-resolution (SR) technology is widely used in a variety of fields, including gaming, film, cameras, and medical imaging, to reconstruct low-resolution images into super-resolution images.

[0003] Currently, super-resolution technology based on deep learning has become mainstream. However, the existing neural network structure built based on deep learning requires a lot of resources for super-resolution reconstruction, and the neural network structure needs to be optimized. Summary of the Invention

[0004] To solve the above technical problems, the embodiments of the present application hope to provide a method, chip, device and electronic device for image processing.

[0005] The technical solution of this application is achieved as follows:

[0006] In a first aspect, a method for image processing is provided, comprising:

[0007] Get the original image;

[0008] Reconstructing the original image using an image processing neural network to obtain a super-resolution image;

[0009] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules;

[0010] The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling modules.

[0011] In a second aspect, an image processing chip is provided, comprising:

[0012] A processor is used to obtain an original image and reconstruct the original image using an image processing neural network to obtain a super-resolution image;

[0013] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules;

[0014] The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling modules.

[0015] In a third aspect, an apparatus for image processing is provided, comprising:

[0016] an acquisition module, configured to acquire an original image; and

[0017] A processing module, configured to reconstruct the original image using an image processing neural network to obtain a super-resolution image;

[0018] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules;

[0019] The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling modules.

[0020] In a fourth aspect, an electronic device is provided, comprising: a processor and a memory configured to store a computer program capable of running on the processor,

[0021] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0022] In a fifth aspect, a computer storage medium is also provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0023] In an embodiment of the present application, a method, chip, device and electronic device for image processing are provided, which utilizes an image processing neural network to reconstruct the original image to obtain a super-resolution image; wherein the image processing neural network is a UNET network searched based on a micro-neural network structure; the first downsampling module and the corresponding first upsampling module in the image processing neural network are connected through a channel attention module, and the channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module. In this way, the UNET network can be used to combine deep-level features and shallow-level features, refine the image, and improve the reconstruction effect of the super-resolution image. Each layer of the channel attention module performs channel learning on the feature map data from the two layers of downsampling modules in the encoder, enhances the attention channel features, realizes the selective reuse of the channel, and further improves the performance of the image processing neural network and the reliability of the image super-resolution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 Schematic diagram of the process of image processing in an embodiment of the present application;

[0025] Figure 2(a) to Figure 2(d) This is a schematic diagram of the first evolution process of the cell structure in the embodiment of the present application;

[0026] Figure 3 This is a schematic diagram of the first component structure of the image processing neural network in an embodiment of the present application;

[0027] Figure 4 This is a schematic diagram of the implementation process of the PixelShuffle operator in an embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of the second component structure of the image processing neural network in an embodiment of the present application;

[0029] Figure 6 Schematic diagram of the composition structure of the channel attention module in the embodiment of this application

[0030] Figure 7 Schematic diagram of the process of constructing an image processing neural network in an embodiment of the present application;

[0031] Figure 8(a) to Figure 8(c) This is a schematic diagram of the second evolution process of the cell structure in the embodiment of the present application;

[0032] Figure 9Schematic diagram of the structure of the downsampling module in the embodiment of the present application;

[0033] Figure 10 Schematic diagram of the composition structure of the upsampling module in an embodiment of the present application;

[0034] Figure 11 Schematic diagram of the composition structure of the image processing chip in the embodiment of the present application;

[0035] Figure 12 Schematic diagram of the structure of the apparatus for image processing in an embodiment of the present application;

[0036] Figure 13 Schematic diagram of the structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below with reference to the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0038] Figure 1 FIG. 1 is a flow chart of a method for image processing in an embodiment of the present application, as shown in FIG. Figure 1 As shown, the method may specifically include:

[0039] Step 101: Obtaining an original image;

[0040] Here, the original image can be a low-resolution image obtained by the electronic device from other electronic devices or network devices. It should be noted that the image processing neural network provided in this application can perform super-resolution reconstruction on a single frame image or on multiple frames of images in a video.

[0041] The original image may also be a low-resolution image obtained by converting a light source signal captured by a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor into a digital signal.

[0042] Step 102: reconstructing the original image using an image processing neural network to obtain a super-resolution image;

[0043] The image processing neural network is a UNET network obtained based on the Differentiable Architecture Search (DARTS) search. The UNET network can combine deep-level features with shallow-level features, refine images, and improve the reconstruction effect of super-resolution images.

[0044] The image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules; the channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another of the at least one downsampling module.

[0045] Here, each layer of channel attention module uses the first feature map data output by the first downsampling module of the current layer and the second feature map data output by the second downsampling module of other layers for channel learning, which can enhance the attention channel features, realize the selective reuse of channels, and further improve the performance of image processing neural network and the reliability of image super-resolution results.

[0046] Here, the second downsampling module may be a downsampling module of a higher layer than the current layer where the first downsampling module is located, or a downsampling module of a lower layer than the current layer. Figure 3 As shown in the figure, the network layers increase from top to bottom, that is, the first layer is the lowest layer and the fourth layer is the highest layer.

[0047] Exemplarily, in some implementations, when the first downsampling module is at the highest layer in the at least one downsampling module, the first feature map data output by the first downsampling module is identity mapped to generate the second feature map data; when the first downsampling module is not at the highest layer in the at least one downsampling module, the channel attention module corresponding to the second downsampling module is used to generate the second feature map data.

[0048] Exemplarily, in some embodiments, the method further includes: searching for an upsampling module and a downsampling module based on a differentiable neural network structure, and constructing the image processing neural network using the downsampling module, the upsampling module, and the channel attention module. Because the image processing neural network includes multiple downsampling modules and upsampling modules with the same structure, the image processing neural network can be constructed by automatically searching for a minimized unit based on DARTS.

[0049] Specifically, the DARTS search process consists of three steps: defining the search space, executing the search strategy, and performing performance evaluation. Specifically, given a set of candidate neural network structures in the search space, a search strategy is executed to identify the network structures, and the performance of the searched network structures is evaluated to determine the optimal network architecture parameters and ultimately obtain the network structure with the best performance.

[0050] Figure 2(a) to Figure 2(d) This is a schematic diagram of the first evolution process of the cell structure in the embodiment of this application. Figure 2(a) to Figure 2(d) The physical meaning of each small rectangle (called a network node) in the figure is a feature map. The first feature map is the input of the cell, and the last feature map is the output of the cell. The lines between the feature maps represent operations. Figure 2(a) defines the search space, the number of network nodes in each cell, and the types of operators in the search space. Based on the operators given in the search space and the initial weight values ​​assigned to each operator, a directed acyclic graph is constructed to represent the structure of the cell. The "?" between two nodes in Figure 2(a) indicates that the operation operator used between the two network nodes is unknown at the beginning of the search. A series of search processes are required to determine the one or more operation operators ultimately used between the two network nodes. In practical applications, the initial weight values ​​of each operator between two nodes can be equal, that is, each operator has an equal probability of being selected. Figure 2(b) shows the set of all operators in the cell. Figure 2(c) shows the weights of each operator after training. Thicker lines indicate larger weights and a greater likelihood of selection, while thinner lines indicate smaller weights and a lower likelihood of selection. After all parameters converge, the operator with the largest weight between each pair of nodes is selected as the operator in the final cell architecture. Figure 2(d) shows the final selected cell architecture. It can be seen that the operator with the largest weight is selected, i.e., the line between the current network node and the previous network node with the largest weight.

[0051] Then any network node in the middle can be expressed by the formula:

[0052]

[0053] Where i and j represent the sequence numbers of the network nodes, x is the feature map represented by the network node, and o is the operator between two network nodes. The above formula means that all the previous network nodes are accumulated after the operator is used as the current network node.

[0054] Figure 3 This is a schematic diagram of the first component structure of the image processing neural network in the embodiment of the present application. Figure 3As shown, the image processing neural network is divided into two parts: the encoder on the left and the decoder on the right. The encoder includes a downsampling module, and the decoder includes an upsampling module. The original image is downsampled through the encoder's four layers of downsampling modules to extract some superficial image features. The feature map data output by the highest-level downsampling module undergoes identity mapping and is input into the corresponding upsampling module. It then passes through the decoder's four layers of upsampling modules for upsampling to extract some deeper features. Each layer of downsampling modules is connected to the upsampling modules via a channel attention (CA) module. Each CA layer performs channel learning on the first feature map data output by the first downsampling module in the current encoder layer and the second feature map data output by the second downsampling module in the next higher layer to extract the channel attention feature map. For the highest layer, the second feature map data is generated by the identity mapping of the first feature map data. Each layer of CA performs channel learning on the feature map data output by the two layers of downsampling modules to enhance the attention to the channel features, realizes the selective reuse of the channels, and sends the channel attention feature map data output by the channel attention module to the corresponding upsampling module in the decoder to participate in the network calculation of the current layer, which can improve the performance of the image processing neural network and the reliability of the image super-resolution results.

[0055] The image processing neural network also includes an input module and an output module; the input module is used to reduce the size of the input feature map by two times, and the output module is used to increase the size of the input feature map by two times, as well as up-sampling by r times.

[0056] like Figure 3 As shown in the figure, the input module is located before the first layer downsampling module, and the output module is located after the first layer upsampling module. At the head of the UNET network, the input module is composed of a convolution operator with a stride of 2 (Convolution with stride = 2) and an activation function (Leaky ReLU). At the tail of the UNET network, the output module is composed of a deconvolution operator with a stride of 2 (Transpose Convolution with stride = 2), an activation function (Leaky ReLU) and a pixel shuffle operator (PixelShuffle), which can achieve a two-fold reduction and a two-fold increase in the size of the input feature map. The PixelShuffle operator can achieve r 2 The feature map of size ×h×w is mapped to the feature map of size rw×rh, thereby achieving r-fold upsampling of the image.

[0057] Figure 4Schematic diagram of the implementation process of the PixelShuffle operator in the embodiment of the present application. Figure 4 As shown in the figure, a H×W low-resolution input image (Low Resolution) is converted into a rHxrW high-resolution image (High Resolution) through PixelShuffle operation. However, the implementation process does not directly generate this high-resolution image through interpolation or other methods, but first obtains r through convolution. 2 The feature map of each channel is obtained (the size of the feature map is consistent with the size of the input low-resolution image), and then a high-resolution image is output through the periodic shuffling method, where r is the upscaling factor, that is, the image enlargement ratio.

[0058] Based on the above image processing neural network, the image processing neural network structure is further illustrated with examples.

[0059] Exemplarily, in some embodiments, a fusion module is further included between the first downsampling module and the corresponding first upsampling module; the fusion module is used to fuse the first feature map data and the channel attention feature map data output by the channel attention module, and input the fused feature map data into the first upsampling module.

[0060] Figure 5 This is a schematic diagram of the second component structure of the image processing neural network in the embodiment of the present application. Figure 5As shown, the image processing neural network is divided into two parts: the encoder on the left and the decoder on the right. The encoder includes a downsampling module, and the decoder includes an upsampling module. The original image is downsampled through the encoder's four layers of downsampling modules to extract some superficial image features. The feature map data output by the last layer of downsampling modules undergoes identity mapping and is input into the last layer of upsampling modules. It is then upsampled through the decoder's four layers of upsampling modules to extract some deeper features. Each layer of downsampling modules is connected to the upsampling modules by a channel attention (CA) module. Each CA layer performs channel learning on the first feature map data output by the first downsampling module in the current encoder layer and the second feature map data output by the second downsampling module in the next higher layer to extract the channel attention feature map. For the highest layer, the second feature map data is generated by the first feature map data through identity mapping. The channel attention feature map data output by CA is then fused with the first feature map data. The fused channel attention feature map data is fed into the upsampling module in the decoder to participate in the network calculation of the current layer to improve the reconstruction effect of the super-resolution image. The UNET network can combine deep and shallow features to refine the image.

[0061] Based on the above image processing neural network, the channel attention module structure is further illustrated.

[0062] In some embodiments, the channel attention module includes: a 3*3 convolution layer, a 1*1 convolution layer, a global pooling layer and a multiplication layer; the second feature map data passes through the global pooling layer and the 1*1 convolution layer in sequence to obtain the channel attention; the first feature map data passes through the 3*3 convolution layer for convolution operation, and the convolution operation result and the channel attention are multiplied by channel through the multiplication layer to obtain the channel attention feature map data.

[0063] In some embodiments, the channel attention module includes: an upsampling layer and an addition layer; the second feature map data is added through the upsampling layer and the attention feature map data output by the multiplication layer through the addition layer to obtain channel attention feature map data.

[0064] For example, Figure 6 This is a schematic diagram of the composition structure of the channel attention module in the embodiment of the present application. Figure 6As shown, the channel attention module includes two inputs, namely the first feature map data and the second feature map data. The second feature map data is sequentially passed through the global pooling layer (global poling) and the 1*1 convolution layer (1*1conv) to obtain the channel attention. After the first feature map data is convolved by the 2*2 convolution layer (2*2conv), it is multiplied with the channel attention by channel to obtain the channel attention feature map data, and then added to the second feature map data after the upsampling layer (upsampling) through the addition layer to obtain the channel attention feature map data finally output by the channel attention module.

[0065] The UNET network, a neural network used for image processing, combines deep and shallow features, refines images, and improves super-resolution image reconstruction. Each layer's channel attention module performs channel learning on feature map data from the encoder, focusing on channel features and enabling selective reuse of channels. This further improves the performance of the image processing neural network and the reliability of image super-resolution results.

[0066] In practical applications, the operator with the largest weight is usually selected in the DARTS search, and all other operators are discarded. This approach will bring about large errors in network performance when the difference in operator weights is not obvious, and there is a problem of suboptimal network performance. In view of this, the embodiment of the present application further optimizes the network search based on sharpness-aware minimization (SAM) to solve the problem of suboptimal network performance. The UNET network includes weight parameters and architecture parameters. The two sets of parameters are unified in a framework and optimized by SAM. The optimization method based on SAM can make the loss function convergence area during network training more gentle, which is conducive to improving the generalization performance of the model.

[0067] Based on the above embodiments, the construction of the image processing neural network is further illustrated. Figure 7 This is a flow chart of the method for constructing an image processing neural network in an embodiment of the present application. Figure 7 As shown, the method specifically includes:

[0068] Step 701: searching for a downsampling module and an upsampling module based on a differentiable neural network structure;

[0069] The downsampling module includes a first network node, at least two operators are included between the first network node and a predecessor network node, and the output of the predecessor network node is fused after passing through the at least two operators to obtain the input of the first network node;

[0070] The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the outputs of the previous network node are respectively fused after passing through the at least two operators to obtain the input of the second network node.

[0071] For example, when at least two operators are selected between two network nodes, operator-level fusion is performed by averaging the feature maps output by all operators. Retaining multiple valid operators in the upsampling and downsampling modules and performing operator-level fusion can address the problem of suboptimal models caused by unclear differences in operator weights, reducing network computation while ensuring network performance.

[0072] Exemplarily, in some embodiments, the first network node and all predecessor network nodes include at least two operators, or the first network node and a portion of the predecessor network nodes include at least two operators, and the first network node and another portion of the predecessor network nodes include one operator. The second network node and all predecessor network nodes include at least two operators, or the second network node and a portion of the predecessor network nodes include at least two operators, and the second network node and another portion of the predecessor network nodes include one operator. In some embodiments, the downsampling module and the upsampling module may further include other network nodes, and the other network nodes and the predecessor network node include one operator. It can be understood that when the weight difference of the operators is large, only the operator with the largest weight can be selected between the two network nodes, and the other operators can be discarded.

[0073] Exemplarily, in some embodiments, searching for a downsampling module and an upsampling module based on a micro-neural network structure includes:

[0074] pre-define a first search space and a second search space;

[0075] Searching the first search space based on a network parameter optimization strategy to obtain the downsampling module;

[0076] Search the second search space based on a network parameter optimization strategy to obtain the upsampling module;

[0077] Among them, the first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2.

[0078] That is to say, when searching for downsampling modules and upsampling modules, different search spaces are pre-defined according to needs, including the number of network nodes and the types of operators; the edges between network nodes are regarded as a mixture of all operators, and the alternating optimization method is used to learn the parameters of the nodes and the weights of the operators. After the optimization is completed, the parameters and operators of the network nodes are determined according to the weights.

[0079] Exemplarily, for the downsampling module, the first search space selects 4 network nodes, multiple downsampling operators, and multiple first identity mapping operators. For the upsampling module, the second search space selects 4 network nodes, multiple upsampling operators, and multiple second identity mapping operators. The downsampling operator is a convolution operator (such as Conv+BN+LReLU operator), and the upsampling operator is a deconvolution operator (such as Transpose_Conv+BN+LReLU operator). The selection of the first identity mapping operator and the second identity mapping operator can be exactly the same or different. The identity mapping operator keeps the size of the feature map unchanged before and after mapping. For example, the identity mapping operator can be conv+BN+LReLU operator.

[0080] The network parameter optimization strategy is used to retain multiple operators between network nodes to solve the problem of suboptimal network model. The network parameter optimization strategy includes one of the following:

[0081] The first optimization strategy selects the top k operators with the largest weights;

[0082] The second optimization strategy is to select operators whose weights have an increasing trend;

[0083] The third optimization strategy is to select an operator with an increasing weight trend from the first k operators with the largest weights; where k is an integer greater than 1.

[0084] For example, a one-level or two-level optimization approach within a UNET network is employed to simultaneously update and optimize the weight parameters and network parameters of the operators in the upsampling and downsampling modules until all parameters converge. Finally, the top three operators with the largest weight parameters, or those with increasing weight parameters, are selected, and all unselected operators are eliminated. After these operators are selected, the output feature maps of all operators are averaged to complete operator-level fusion.

[0085] Figure 8(a) to Figure 8(c) This is a schematic diagram of the second evolution process of the cell structure in the embodiment of this application. Figure 8(a) to Figure 8(c)The physical meaning of each node (called a network node) is a feature map. The first feature map is the input of the cell, and the last feature map is the output of the cell. The lines between the feature maps represent the operation operators. Figure 8(a) defines the search space, the number of network nodes in each cell, the operator types in the search space, and the three operators between two network nodes. Based on the operators given in the search space, each operator is assigned an initial weight value, and the weight of each operator is represented by the thickness of the line. In practical applications, the initial weight value of each operator between two nodes can be equal, that is, the probability of each operator being selected is equal. Figure 8(b) shows the weights of all operators in a cell updated using a gradient descent optimization strategy. The thickness of the line represents the weight of each operator (indicating its likelihood of selection). Thicker lines indicate larger parameter weights and a greater likelihood of selection, while thinner lines indicate smaller parameter weights and a lower likelihood of selection. After all parameters converge, operators with ascending architectural parameters (AS) and descending architectural parameters (DS) are selected. Figure 8(c) shows the final selected cell architecture after training. The AS operator is retained, while the DS operator is discarded. Two operators are retained between Node-1 and Node-2, two operators are retained between Node-3 and Node-4, and one operator is retained between other nodes.

[0086] For example, in the process of searching for the downsampling module and the upsampling module, the objective function for searching for the downsampling module and the upsampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2; where L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term; L edge is the image edge information loss term.

[0087] Here, L1 is specifically the 1-norm loss term between the network output and the given true value, and L2 is specifically the 2-norm loss term between the network output and the given true value.

[0088] In order to constrain computing energy consumption, the above objective function is used to make the searched network model more lightweight, while also taking into account accuracy and efficiency, making it easier to deploy the network on mobile devices such as mobile phones, which can effectively reduce the computing energy consumption of mobile devices.

[0089] Exemplarily, in the process of searching for downsampling modules and upsampling modules, the performance metrics used to search for downsampling modules and upsampling modules based on the differentiable neural network structure include: Peak Signal Noise Ratio (PSNR) and Structural Similarity (SSIM). Here, PSNR and SSIM are used as image quality evaluation indicators. Larger values ​​indicate smaller image distortion and better network performance; conversely, lower network performance.

[0090] Figure 9 This is a schematic diagram of the structure of the downsampling module in the embodiment of the present application. Figure 9 As shown in the figure, the searched downsampling module includes two input nodes (Input0 and Input1) and one output node (Output). D[XXX] represents the fusion of downsampling operators XXX, where X is the identifier of each downsampling operator and can be 0, 1, 2, or 3. N[XXX] represents the fusion of identity mapping operators XXX, where X is the identifier of each identity mapping operator and can be 0, 2, or 3. At the output, the concatenation node (Concat) performs channel concatenation to produce the output feature map.

[0091] Figure 10 This is a schematic diagram of the composition structure of the upsampling module in the embodiment of the present application. Figure 10 As shown in the figure, the searched upsampling module includes two input nodes (Input0 and Input1) and one output node (Output). U[XXX] represents the fusion of upsampling operators XXX, where X is the identifier of each upsampling operator and takes the value 0 or 1. N[XXX] represents the fusion of identity mapping operators XXX, where X is the identifier of each identity mapping operator and takes the value 0, 2, or 3. At the output, the concatenation node (Concat) is used to perform channel concatenation to obtain the output feature map.

[0092] Step 702: constructing the image processing neural network using the downsampling module, the upsampling module, and the channel attention module;

[0093] For example, Figure 3 and Figure 5 As shown in the figure, the UNET network contains 4 layers, each layer contains the same downsampling module, upsampling module and channel attention module. It only needs to automatically search for the minimum unit based on DARTS to build an image processing neural network.

[0094] Exemplarily, in some embodiments, the image processing neural network further includes an input module and an output module; the input module is used to reduce the size of the input feature map by a factor of two, and the output module is used to increase the size of the input feature map by a factor of two, as well as upsampling by r times.

[0095] Step 703: Obtain a training data set.

[0096] Step 704: retrain the image processing neural network using the training data set until the loss function meets the convergence condition, thereby obtaining a trained image processing neural network.

[0097] That is to say, after DARTS completes the automatic search, it builds an image processing neural network based on all retained operators, and retrains the image processing neural network using the training data to obtain the final applicable image processing neural network.

[0098] During the search process, this image processing neural network retains multiple valid operators and performs operator-level fusion. This solves the problem of suboptimal models caused by unclear differences in operator weights. This reduces network computational effort while ensuring network performance, making the network easier to deploy on mobile devices such as mobile phones. This effectively reduces the computational energy consumption of mobile devices while maintaining network performance. Introducing the channel attention module into the UNET network results in a more optimized image processing neural network, improving image super-resolution.

[0099] The present application also provides an image processing chip. Figure 11 As shown, the image processing chip 110 includes:

[0100] Processor 1101 is used to obtain an original image, and perform super-resolution reconstruction on the original image using an image processing neural network to obtain a super-resolution image;

[0101] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules;

[0102] The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling modules.

[0103] In some embodiments, when the first downsampling module is at the highest layer in the at least one downsampling module, the first feature map data output by the first downsampling module is identity mapped to generate the second feature map data; when the first downsampling module is not at the highest layer in the at least one downsampling module, the channel attention module corresponding to the second downsampling module is used to generate the second feature map data.

[0104] In some embodiments, the channel attention module includes: a 3*3 convolution layer, a 1*1 convolution layer, a global pooling layer and a multiplication layer; the second feature map data passes through the global pooling layer and the 1*1 convolution layer in sequence to obtain the channel attention; the first feature map data passes through the 3*3 convolution layer for convolution operation, and the convolution operation result and the channel attention are multiplied by channel through the multiplication layer to obtain the channel attention feature map data.

[0105] In some embodiments, the channel attention module further includes: an upsampling layer and an addition layer; the second feature map data is added through the upsampling layer and the attention feature map data output by the multiplication layer through the addition layer to obtain channel attention feature map data.

[0106] In some embodiments, a fusion module is further included between the first downsampling module and the corresponding first upsampling module; the fusion module is configured to fuse the first feature map data with the channel attention feature map data output by the channel attention module, and input the fused feature map data into the first upsampling module. Exemplarily, the fusion module concatenates the first feature map data output by the first downsampling module and the channel attention feature map data output by the channel attention module, or the fusion module directly feeds the two feature map data into the second upsampling module.

[0107] In some embodiments, the image processing neural network further includes an input module and an output module; wherein the input module is used to reduce the size of the input feature map by a factor of two, and the output module is used to increase the size of the input feature map by a factor of two, as well as upsampling by a factor of r.

[0108] In some embodiments, the downsampling module and the upsampling module are searched based on a differentiable neural network structure;

[0109] The downsampling module includes a first network node, at least two operators are included between the first network node and a predecessor network node, and the output of the predecessor network node is fused after passing through the at least two operators to obtain the input of the first network node;

[0110] The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the outputs of the previous network node are respectively fused after passing through the at least two operators to obtain the input of the second network node.

[0111] In this way, during the search process, retaining multiple valid operators and performing operator-level fusion can solve the problem of suboptimal model caused by unclear differences in operator weights, and reduce the amount of network calculation while ensuring network performance.

[0112] In some embodiments, when the downsampling module and the upsampling module are searched based on the micro-neural network structure, a first search space and a second search space are predefined; the first search space is searched based on a network parameter optimization strategy to obtain the downsampling module; the second search space is searched based on a network parameter optimization strategy to obtain the upsampling module;

[0113] Among them, the first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2.

[0114] In some embodiments, the network parameter optimization strategy includes: a first optimization strategy, selecting the first k operators with the largest weights; where k is an integer greater than 1; or a second optimization strategy, selecting operators whose weights have an increasing trend.

[0115] In some embodiments, the objective function for searching the down-sampling module and the up-sampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2;

[0116] Among them, L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term, L edge is the image edge information loss item;

[0117] In some embodiments, performance metrics for the downsampling module and the upsampling module searched based on the differentiable neural network structure include: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0118] In some embodiments, the processor 1101 includes at least one of the following: a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural-network processing unit (NPU).

[0119] The image processing chip 110 may further include a memory 1102 for storing a computer program that can be run on the processor 1101 . The processor 1101 is configured to perform the aforementioned super-resolution image reconstruction operation when running the computer program.

[0120] The above-mentioned memory can be a volatile memory (volatile memory), such as a random-access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above types of memory, and provides instructions and data to the processor.

[0121] Optionally, the image processing chip 110 may further include an input interface 1103. The input interface 1103 is used to communicate with other devices or chips, and specifically, to obtain information or data sent by other devices or chips.

[0122] Optionally, the image processing chip 110 may further include an output interface 1104. The output interface 1104 is used to communicate with other devices or chips, and specifically, information or data may be output to other devices or chips.

[0123] Optionally, the image processing chip 110 may be applied to the electronic device in the embodiment of the present application.

[0124] Of course, in actual applications, the various components in the chip are coupled together via a bus system. It is understood that the bus system is used to enable communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0125] It should be noted that the above-mentioned image processing neural network in the embodiment of the present application can also be deployed on any other hardware computing unit for implementation.

[0126] The present application also provides an apparatus for image processing, such as Figure 12 As shown, the device 120 includes:

[0127] An acquisition module 1201 is configured to acquire an original image; and

[0128] A processing module 1202 is configured to reconstruct the original image using an image processing neural network to obtain a super-resolution image;

[0129] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules;

[0130] The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling modules.

[0131] In some embodiments, the processing module 1202 is used to perform identity mapping on the first feature map data output by the first downsampling module to generate the second feature map data when the first downsampling module is at the highest layer in the at least one downsampling module; when the first downsampling module is not at the highest layer in the at least one downsampling module, use the channel attention module corresponding to the second downsampling module to generate the second feature map data.

[0132] In some embodiments, the channel attention module includes: a 3*3 convolution layer, a 1*1 convolution layer, a global pooling layer and a multiplication layer; the second feature map data passes through the global pooling layer and the 1*1 convolution layer in sequence to obtain the channel attention; the first feature map data passes through the 3*3 convolution layer for convolution operation, and the convolution operation result and the channel attention are multiplied by channel through the multiplication layer to obtain the channel attention feature map data.

[0133] In some embodiments, the channel attention module further includes: an upsampling layer and an addition layer; the second feature map data is added through the upsampling layer and the attention feature map data output by the multiplication layer through the addition layer to obtain channel attention feature map data.

[0134] In some embodiments, a fusion module is further included between the first downsampling module and the corresponding first upsampling module; the fusion module is used to fuse the first feature map data and the channel attention feature map data output by the channel attention module, and input the fused feature map data into the first upsampling module.

[0135] In some embodiments, the image processing neural network further includes an input module and an output module; wherein the input module is used to reduce the size of the input feature map by a factor of two, and the output module is used to increase the size of the input feature map by a factor of two, as well as upsampling by a factor of r.

[0136] In some embodiments, the downsampling module and the upsampling module are searched based on a differentiable neural network structure;

[0137] The downsampling module includes a first network node, at least two operators are included between the first network node and a predecessor network node, and the output of the predecessor network node is fused after passing through the at least two operators to obtain the input of the first network node;

[0138] The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the outputs of the previous network node are respectively fused after passing through the at least two operators to obtain the input of the second network node.

[0139] In this way, during the search process, retaining multiple valid operators and performing operator-level fusion can solve the problem of suboptimal model caused by unclear differences in operator weights, and reduce the amount of network calculation while ensuring network performance.

[0140] In some embodiments, when the downsampling module and the upsampling module are searched based on the micro-neural network structure, a first search space and a second search space are predefined; the first search space is searched based on a network parameter optimization strategy to obtain the downsampling module; the second search space is searched based on a network parameter optimization strategy to obtain the upsampling module;

[0141] Among them, the first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2.

[0142] In some embodiments, the network parameter optimization strategy includes: a first optimization strategy, selecting the first k operators with the largest weights; where k is an integer greater than 1; or a second optimization strategy, selecting operators whose weights have an increasing trend.

[0143] In some embodiments, the objective function for searching the down-sampling module and the up-sampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2;

[0144] Among them, L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term, L edge is the image edge information loss item;

[0145] In some embodiments, the performance metrics used to search for downsampling modules and upsampling modules based on the differentiable neural network structure include: peak signal-to-noise ratio (PSNR) and structural similarity (SSIM).

[0146] For example, at the head and tail of the UNET network, the input module is composed of a convolution operator with stride=2 and an activation function (Leaky ReLU). The output module is composed of a deconvolution operator with stride=2, an activation function (Leaky ReLU) and a pixel shuffle operator, which can achieve a two-fold decrease and a two-fold increase in the size of the input feature map. The PixelShuffle operator can realize the r 2 The feature map of size ×h×w is mapped to the feature map of size rw×rh, thereby achieving r-fold upsampling of the image.

[0147] The above-mentioned device for image processing can be an electronic device or a chip used in an electronic device. In this application, the device can implement the functions of multiple units through software, hardware, or a combination of software and hardware, so that the device can execute the control method provided in any of the above embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the control method, and this application will not elaborate on them one by one.

[0148] The present application also provides an electronic device, such as Figure 13 As shown, the electronic device 130 includes: a processor 1301 and a memory 1302 configured to store a computer program that can be run on the processor 1301.

[0149] The processor 1301 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0150] The processor 1301 includes at least one of the following: a central processing unit (CPU); a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural network processor (NPU).

[0151] Of course, in actual application, Figure 13 As shown, the various components in the electronic device are coupled together via a bus system. It is understood that the bus system is used to enable communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus, and a status signal bus.

[0152] The electronic devices described in this application may include, for example, mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), wearable devices, smart bracelets, cameras, etc.

[0153] In an exemplary embodiment, the present application also provides a computer-readable storage medium, such as a memory including a computer program, which can be executed by a processor of an electronic device to complete the steps of the aforementioned method.

[0154] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit this application. The singular forms of "a", "said" and "the" used in this application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "can have", "including" and "comprising", or "can include" and "can include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but do not exclude the presence of additional features.

[0155] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various types of information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another and are not necessarily used to describe a specific order or precedence. For example, first information could be referred to as second information, and similarly, second information could be referred to as first information without departing from the scope of the present invention.

[0156] The technical solutions described in the embodiments of this application can be combined arbitrarily unless there is any conflict.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are merely illustrative. For example, the division of units is merely a logical function division. In actual implementation, there may be other division methods, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0158] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0159] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above-mentioned integrated units can be implemented in the form of hardware or in the form of hardware plus software functional units.

[0160] The above is only a specific implementation method of the present application, but the scope of protection of the present application is not limited thereto. Any technician familiar with this technical field can easily think of changes or replacements within the technical scope disclosed in this application, which should be covered by the scope of protection of the present application.

Claims

1. A method for image processing, characterized in that: The method comprises: Get the original image; Reconstructing the original image using an image processing neural network to obtain a super-resolution image; The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling module, and the first upsampling module is one of the at least one upsampling module; The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling module; The method further comprises: Searching for the down-sampling module and the up-sampling module based on a differentiable neural network structure; Using the downsampling module, the upsampling module and the channel attention module, construct the image processing neural network; The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node; The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

2. The method according to claim 1, characterized in that The method further comprises: When the first down-sampling module is at the highest level in the at least one down-sampling module, performing identity mapping on the first feature map data output by the first down-sampling module to generate the second feature map data; When the first down-sampling module is not at the highest layer in the at least one down-sampling module, the second feature map data is generated using a channel attention module corresponding to the second down-sampling module.

3. The method according to claim 1, characterized in that The channel attention module includes: a 3*3 convolution layer, a 1*1 convolution layer, a global pooling layer and a multiplication layer; The second feature map data passes through the global pooling layer and the 1*1 convolution layer in sequence to obtain channel attention; The first feature map data is convolved through a 3*3 convolutional layer, and the convolution operation result and the channel attention are multiplied by channel through the multiplication layer to obtain the channel attention feature map data.

4. The method according to claim 3, characterized in that The channel attention module also includes: an upsampling layer and an addition layer; The second feature map data is added through the upsampling layer and the attention feature map data output by the multiplication layer through the addition layer to obtain channel attention feature map data.

5. The method according to claim 1, characterized in that A fusion module is also included between the first down-sampling module and the corresponding first up-sampling module; The fusion module is used to fuse the first feature map data with the channel attention feature map data output by the channel attention module, and input the fused feature map data into the first upsampling module.

6. The method according to any one of claims 1 to 5, characterized in that: The image processing neural network also includes an input module and an output module; The input module is used to reduce the size of the input feature map by two times, and the output module is used to increase the size of the input feature map by two times, and upsample by r times.

7. The method according to claim 1, characterized in that The searching for the down-sampling module and the up-sampling module based on the micro-neural network structure includes: pre-define a first search space and a second search space; Search the first search space based on a network parameter optimization strategy to obtain the downsampling module; Search the second search space based on a network parameter optimization strategy to obtain the upsampling module; The first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2; The network parameter optimization strategy includes one of the following: The first optimization strategy selects the first k operators with the largest weights; The second optimization strategy is to select operators whose weights have an upward trend; The third optimization strategy is to select an operator whose weight has an increasing trend from the first k operators with the largest weights; wherein k is an integer greater than 1.

8. The method according to claim 6, characterized in that The objective function for the down-sampling module and the up-sampling module searched based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2; Among them, L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term, L edge is the image edge information loss item; The performance metrics for the down-sampling module and the up-sampling module searched based on the differentiable neural network structure include: peak signal-to-noise ratio PSNR and structural similarity SSIM.

9. An image processing chip, characterized in that: include: A processor, used to acquire an original image, and reconstruct the original image using an image processing neural network to obtain a super-resolution image; The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling module, and the first upsampling module is one of the at least one upsampling module; The channel attention module is configured to extract channel attention feature map data using the first feature map data output by the first downsampling module and the second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling module; The down-sampling module and the up-sampling module are searched based on a differentiable neural network structure; The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node; The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

10. The image processing chip according to claim 9, characterized in that: When the first down-sampling module is at the highest level in the at least one down-sampling module, performing identity mapping on the first feature map data output by the first down-sampling module to generate the second feature map data; When the first down-sampling module is not at the highest layer in the at least one down-sampling module, the second feature map data is generated using a channel attention module corresponding to the second down-sampling module.

11. A device for image processing, characterized in that: include: An acquisition module, used for acquiring the original image; as well as A processing module, used to reconstruct the original image using an image processing neural network to obtain a super-resolution image; The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein the first downsampling module and the corresponding first upsampling module are connected through a channel attention module, the first downsampling module is one of the at least one downsampling module, and the first upsampling module is one of the at least one upsampling module; The channel attention module is configured to extract channel attention feature map data using first feature map data output by the first downsampling module and second feature map data output by the second downsampling module, wherein the second downsampling module is another one of the at least one downsampling module; The down-sampling module and the up-sampling module are searched based on a differentiable neural network structure; The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node; The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

12. An electronic device, characterized in that: include: a processor and a memory configured to store a computer program capable of being executed on said processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 8 when running the computer program.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images