A method, chip, device and electronic device for image processing

Abstract

The embodiment of the present application discloses a method, chip, device and electronic device for image processing, which uses an image processing neural network to reconstruct the original image to obtain a super-resolution image; wherein the image processing neural network is a UNET network obtained based on a differentiable neural network structure search, wherein the first downsampling module and the corresponding first upsampling module are connected through a shuffled attention module. In this way, when the UNET network is used for image reconstruction, it is possible to combine deep-level features with shallow-level features, refine the image, and improve the reconstruction effect of the super-resolution image. Each layer of the shuffled attention module performs channel attention and spatial attention learning on the feature map data from the encoder, realizes selective reuse of the channel dimension and the spatial dimension to enhance the attention feature, further optimizes the image processing neural network, and improves the network performance and the reliability of the image super-resolution results.

Description

Technical Field

[0001] The present application relates to image processing technology, and in particular to a method, chip, device and electronic device for image processing. Background Art

[0002] With the popularity of high-definition devices, the resolution of user-side display devices has generally increased to 2K or even higher. Image super-resolution (SR) technology is widely used in many fields such as games, movies, cameras, and medical imaging to achieve the reconstruction of low-resolution images to super-resolution images.

[0003] At present, super-resolution technology based on deep learning has become mainstream. However, the existing neural network structure built based on deep learning requires a lot of resources for super-resolution reconstruction, and the neural network structure needs to be optimized. Summary of the invention

[0004] In order to solve the above technical problems, the embodiments of the present application hope to provide a method, chip, device and electronic device for image processing.

[0005] The technical solution of this application is implemented as follows:

[0006] In a first aspect, a method for image processing is provided, comprising:

[0007] Get the original image;

[0008] Reconstructing the original image using an image processing neural network to obtain a super-resolution image;

[0009] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules;

[0010] Wherein, the shuffled attention module is configured to: perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module.

[0011] In a second aspect, an image processing chip is provided, comprising:

[0012] A processor, configured to acquire an original image, perform super-resolution reconstruction on the original image using an image processing neural network, and output a super-resolution image;

[0013] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules;

[0014] Wherein, the shuffled attention module is configured to: perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module.

[0015] In a third aspect, a device for image processing is provided, comprising:

[0016] An acquisition module, used for acquiring an original image; and

[0017] A processing module, used to reconstruct the original image using an image processing neural network to obtain a super-resolution image;

[0018] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules;

[0019] Wherein, the shuffled attention module is configured to: perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module.

[0020] In a fourth aspect, an electronic device is provided, comprising: a processor and a memory configured to store a computer program that can be run on the processor,

[0021] Wherein, the processor is configured to execute the steps of the aforementioned method when running the computer program.

[0022] In a fifth aspect, a computer storage medium is also provided, on which a computer program is stored, wherein the computer program implements the steps of the aforementioned method when executed by a processor.

[0023] In an embodiment of the present application, a method, chip, device and electronic device for image processing are provided, wherein the method uses an image processing neural network to reconstruct the original image to obtain a super-resolution image; wherein the image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the first downsampling module and the corresponding first upsampling module in the image processing neural network are connected through a shuffled attention module, and the shuffled attention module is configured to: perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module. In this way, when the UNET network is used for image reconstruction, it is possible to achieve a combination of deep-level features and shallow-level features, refine the image, and improve the reconstruction effect of the super-resolution image. Each layer of the shuffled attention module performs channel attention and spatial attention learning on the feature map data from the encoder, realizes selective reuse of the channel dimension and the spatial dimension to enhance the attention feature, further optimizes the image processing neural network, and improves the network performance and the reliability of the image super-resolution results. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 A schematic diagram of a process for image processing in an embodiment of the present application;

[0025] Figure 2(a) to Figure 2(d) This is a schematic diagram of the first evolution process of the cell structure in the embodiment of the present application;

[0026] Figure 3 Schematic diagram of the composition structure of the image processing neural network in the embodiment of the present application;

[0027] Figure 4 Schematic diagram of the composition structure of the shuffle attention module in the embodiment of the present application;

[0028] Figure 5 This is a schematic diagram of the implementation process of the PixelShuffle operator in the embodiment of the present application;

[0029] Figure 6 A schematic diagram of the process of constructing an image processing neural network in an embodiment of the present application;

[0030] Figure 7(a) to Figure 7(c) This is a schematic diagram of the second evolution process of the cell structure in the embodiment of the present application;

[0031] Figure 8 Schematic diagram of the structure of the downsampling module in the embodiment of the present application;

[0032] Fig. 9 Schematic diagram of the composition structure of the upsampling module in the embodiment of the present application;

[0033] Fig.10 Schematic diagram of the composition structure of the image processing chip in the embodiment of the present application;

[0034] Fig.11 A schematic diagram of the structure of an apparatus for image processing in an embodiment of the present application;

[0035] Fig.12 It is a schematic diagram of the composition structure of the electronic device in the embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to enable a more detailed understanding of the features and technical contents of the embodiments of the present application, the implementation of the embodiments of the present application is described in detail below in conjunction with the accompanying drawings. The attached drawings are for reference only and are not used to limit the embodiments of the present application.

[0037] Figure 1 is a flow chart of a method for image processing in an embodiment of the present application, such as Figure 1 As shown, the method may specifically include:

[0038] Step 101: Obtaining an original image;

[0039] Here, the original image can be a low-resolution image obtained by the electronic device from other electronic devices or network devices. It should be noted that the image processing neural network provided in this application can perform super-resolution reconstruction for a single frame image or for multiple frames of images in a video.

[0040] The original image may also be a low-resolution image obtained by converting a light source signal captured by a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge coupled device (CCD) image sensor into a digital signal.

[0041] Step 102: reconstructing the original image using an image processing neural network to obtain a super-resolution image;

[0042] The image processing neural network is a UNET network obtained based on the Differentiable Architecture Search (DARTS) search. The UNET network can be used to combine deep-level features with shallow-level features, refine images, and improve the reconstruction effect of super-resolution images.

[0043] The image processing neural network includes at least one downsampling module and at least one upsampling module corresponding to the at least one downsampling module, wherein a first downsampling module and a corresponding first upsampling module are connected through a shuffled attention module, the first downsampling module is one of the at least one downsampling modules, and the first upsampling module is one of the at least one upsampling modules; wherein the shuffled attention module is configured to perform channel shuffling operations and operator fusion operations on feature map data output by the first downsampling module.

[0044] Exemplarily, the shuffled attention module is further configured to extract channel attention feature map data and spatial attention feature map data of the feature map data output by the first down-sampling module, fuse the channel attention feature map data and the spatial attention feature map data, and input them into the first up-sampling module, and participate in the network calculation of the current layer together with the feature map data output by the first up-sampling module.

[0045] Here, the shuffled attention module is used to perform channel attention and spatial attention learning, so as to achieve selective reuse of channel dimension and spatial dimension to enhance the attention features, further improve the performance of image processing neural network and the reliability of image super-resolution results.

[0046] Exemplarily, in some embodiments, the method further includes: searching for an upsampling module and a downsampling module based on a differentiable neural network structure, and constructing the image processing neural network using the downsampling module, the upsampling module, and the shuffled attention module. Since the image processing neural network contains multiple downsampling modules and upsampling modules of the same structure, the image processing neural network can be constructed by only automatically searching for the minimum unit based on DARTS.

[0047] Specifically, the search process of DARTS mainly includes three parts: defining the search space, executing the search strategy, and performance evaluation. Specifically, given a set of candidate neural network structures in a search space, a certain search strategy is executed to search for the network structure, the performance of the searched network structure is evaluated, the optimal network architecture parameters are determined, and the network structure with the best performance is obtained.

[0048] Figure 2(a) to Figure 2(d) This is a schematic diagram of the first evolution process of the cell structure in the embodiment of the present application. Figure 2(a) to Figure 2(d)The physical meaning of each small rectangle (called a network node) is a feature map. The first feature map is the input of the cell, and the last feature map is the output of the cell. The lines between the feature maps represent the operation operators. Figure 2(a) defines the search space, the number of network nodes in each cell, and the operator types in the search space. Based on the operators given in the search space and the initial weight values ​​assigned to each operator, a directed acyclic graph is constructed to represent the structure of the cell. The "?" between two nodes in Figure 2(a) indicates that the operation operator used between the two network nodes is unknown when the search starts, and a series of search processes are required to determine the one or more operation operators that are ultimately used between the two network nodes. In practical applications, the initial weight values ​​of each operator between two nodes can be equal, that is, each operator has an equal probability of being selected. Figure 2(b) shows the set of all operators in the cell. Figure 2(c) shows the weights of each operator after training. The thicker the line, the greater the weight and the greater the possibility of being selected. The thinner the line, the smaller the weight and the less likely it is to be selected. After all parameters converge, the operator with the largest weight between each pair of nodes is selected as the operator in the final cell architecture. Figure 2(d) shows the final selected cell architecture. It can be seen that the line with the largest weight between the current network node and the previous network node is selected, that is, the operator with the largest weight.

[0049] Then any network node in the middle can be expressed by the formula:

[0050]

[0051] Among them, i and j represent the serial numbers of the network nodes, x is the feature map represented by the network node, and o is the operation operator between two network nodes. The above formula means that all the predecessor network nodes are accumulated as the current network node after the operation operator.

[0052] Figure 3 Schematic diagram of the composition structure of the image processing neural network in the embodiment of the present application, such as Figure 3As shown in the figure, the image processing neural network is divided into two parts: the encoder on the left and the decoder on the right. The encoder includes a down sampling module and the decoder includes an up sampling module. The original image is down sampled by the four down sampling modules of the encoder to extract some superficial image features. The feature map data output by the highest down sampling module is input into the corresponding up sampling module through identity mapping, and then up sampled by the four up sampling modules of the decoder to extract some deep features. Each down sampling module and up sampling module are connected by a shuffle attention (SA) module. Each SA performs channel attention and spatial attention learning on the feature map data from the encoder to achieve selective reuse of channel dimensions and spatial dimensions, and sends the output feature image data to the up sampling module in the decoder for up sampling.

[0053] Exemplarily, in some embodiments, the shuffle attention module includes:

[0054] First segmentation layer, second segmentation layer, channel attention network, spatial attention network, concatenation layer and shuffle fusion layer;

[0055] The first segmentation layer is configured to divide the feature map data output by the first downsampling module into g groups of feature map data according to dimensions;

[0056] The second segmentation layer is configured to generate channel attention feature map data associated with the channel attention network and spatial attention feature map data associated with the spatial attention network based on each set of feature map data;

[0057] The splicing layer is configured to splice the channel attention feature map data and the spatial attention feature map data of each set of feature map data to obtain g sets of spliced ​​feature map data;

[0058] The shuffle fusion layer is configured to perform the channel shuffle operation and the operator fusion operation on the g groups of spliced ​​feature map data to obtain fused feature map data.

[0059] Figure 4 Schematic diagram of the composition structure of the shuffle attention module in the embodiment of the present application. Figure 4As shown in the figure, the shuffled attention module first divides the c channels of the feature map data output by the downsampling module in the encoder into g groups, each with c / g channels, and the following operations are performed on each group separately. For each group, these channels are divided into two parts, each with c / 2g channels. One part is input to the channel attention network for channel attention calculation, and the other part is input to the spatial attention network for spatial attention calculation. The channel attention network uses a structure similar to Squeeze-and-Excitation (SE). The spatial attention network performs a linear operation on the input feature map data after group normalization (GN), and then passes Figure 4 The σ in σ outputs the spatial attention, which is multiplied by the input feature map data to obtain the spatial attention feature map data. Then the outputs of the two attention networks are concatenated to obtain the results of each group. Finally, the results of each group are subjected to channel shuffle and operator fusion operations to obtain the fused feature map data. After that, together with the feature map data output by the upsampling module in the decoder of the UNET network, they participate in the network calculation of the current layer.

[0060] Exemplarily, in some embodiments, the image processing neural network further includes an input module and an output module, wherein the input module is used to reduce the size of the input feature map by a factor of two, and the output module is used to increase the size of the input feature map by a factor of two, as well as upsampling by a factor of r.

[0061] like Figure 3 As shown in the figure, the input module is located before the first layer downsampling module, and the output module is located after the first layer upsampling module. At the head of the UNET network, the input module is composed of a convolution operator with a step size of 2 (Convolution with stride = 2) and an activation function (Leaky ReLU). At the tail of the UNET network, the output module is composed of a deconvolution operator with a step size of 2 (Transpose Convolution with stride = 2), an activation function (Leaky ReLU) and a pixel shuffle operator (PixelShuffle), which can double the size of the input feature map and double the size of the feature map. The PixelShuffle operator can realize the r 2 The feature map of size ×h×w is mapped to a feature map of size rw×rh, thereby achieving r-fold upsampling of the image.

[0062] Figure 5 Schematic diagram of the implementation process of the PixelShuffle operator in the embodiment of the present application. Figure 5As shown in the figure, a H×W low-resolution input image (Low Resolution) is converted into a rHxrW high-resolution image (High Resolution) through the PixelShuffle operation. However, the implementation process does not directly generate this high-resolution image through interpolation or other methods, but first obtains r through convolution. 2 The feature map of each channel is obtained (the size of the feature map is consistent with the size of the input low-resolution image), and then a high-resolution image is output through the periodic shuffling method, where r is the upscaling factor, that is, the image enlargement ratio.

[0063] The above image processing neural network is a UNET network that can realize the combination of deep-level features and shallow-level features, refine the image, and improve the reconstruction effect of super-resolution images. Each layer of shuffled attention module performs channel attention and spatial attention learning on the feature map data from the encoder, realizes the selective reuse of channel dimensions and spatial dimensions to enhance the attention features, further optimizes the image processing neural network, and improves the network performance and the reliability of image super-resolution results.

[0064] In practical applications, the operator with the largest weight is usually selected in the DARTS search, and all other operators are discarded. This approach will bring large errors in network performance when the difference in operator weights is not obvious, and there is a problem of suboptimal network performance. In view of this, the embodiment of the present application further optimizes the network search based on sharpness-aware minimization (SAM) to solve the problem of suboptimal network performance. The UNET network includes weight parameters and architecture parameters. The two sets of parameters are unified in a framework and optimized by SAM. The optimization method based on SAM can make the loss function convergence area during network training more gentle, which is conducive to improving the generalization performance of the model.

[0065] Based on the above embodiments, the construction of the image processing neural network is further illustrated. Figure 6 FIG. 1 is a flow chart of a method for constructing an image processing neural network in an embodiment of the present application. Figure 6 As shown, the method specifically includes:

[0066] Step 601: searching for a downsampling module and an upsampling module based on a differentiable neural network structure;

[0067] The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node;

[0068] The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

[0069] Exemplarily, when at least two operators are selected between two network nodes, the feature maps output by all operators are averaged for fusion at the operator level. Retaining multiple valid operators in the upsampling module and downsampling module and performing operator-level fusion can solve the problem of suboptimal model caused by unclear operator weight differences, and reduce network calculations while ensuring network performance.

[0070] Exemplarily, in some embodiments, the first network node and all predecessor network nodes include at least two operators, or the first network node and a part of the predecessor network nodes include at least two operators, and another part of the predecessor network nodes include one operator. The second network node and all predecessor network nodes include at least two operators, or the second network node and a part of the predecessor network nodes include at least two operators, and another part of the predecessor network nodes include one operator. In some embodiments, the downsampling module and the upsampling module may also include other network nodes, and the other network nodes and the predecessor network nodes include one operator. It can be understood that when the weight difference of the operator is large, only the operator with the largest weight can be selected between the two network nodes, and other operators can be discarded.

[0071] Exemplarily, in some embodiments, searching the down-sampling module and the up-sampling module based on a differentiable neural network structure includes:

[0072] pre-define a first search space and a second search space;

[0073] Search the first search space based on a network parameter optimization strategy to obtain the downsampling module;

[0074] Search the second search space based on a network parameter optimization strategy to obtain the upsampling module;

[0075] Among them, the first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2.

[0076] That is to say, when searching for downsampling modules and upsampling modules, different search spaces are pre-defined according to needs, including the number of network nodes and the types of operators; the edges between network nodes are regarded as a mixture of all operators, and the parameters of the nodes and the weights of the operators are learned by the alternating optimization method. After the optimization is completed, the parameters and operators of the network nodes are determined according to the weights.

[0077] Exemplarily, for the downsampling module, the first search space selects 4 network nodes, multiple downsampling operators and multiple first identity mapping operators. For the upsampling module, the second search space selects 4 network nodes, multiple upsampling operators and multiple second identity mapping operators. The downsampling operator is a convolution operator (such as Conv+BN+LReLU operator), and the upsampling operator is a deconvolution operator (such as Transpose_Conv+BN+LReLU operator). The selection of the first identity mapping operator and the second identity mapping operator can be exactly the same or different, and the identity mapping operator keeps the size of the feature map unchanged before and after mapping, for example, the identity mapping operator can be conv+BN+LReLU operator.

[0078] The network parameter optimization strategy is used to retain multiple operators between network nodes to solve the problem of suboptimal network model. The network parameter optimization strategy includes one of the following:

[0079] The first optimization strategy selects the first k operators with the largest weights;

[0080] The second optimization strategy is to select operators whose weights have an upward trend;

[0081] The third optimization strategy is to select an operator whose weight has an increasing trend from the first k operators with the largest weights; wherein k is an integer greater than 1.

[0082] Exemplarily, the one-level optimization method or two-level optimization method in the UNET network is used to update and optimize the weight parameters and network parameters of the operators in the upsampling module and the downsampling module at the same time until all parameters converge, and finally select the top three operators with the largest weight parameters, or select operators with an increasing weight parameter, and all unselected operators are eliminated. In this way, after selecting these operators, the output feature maps of all operators are averaged to complete the fusion at the operator level.

[0083] Figure 7(a) to Figure 7(c) This is a schematic diagram of the second evolution process of the cell structure in the embodiment of the present application, Figure 7(a) to Figure 7(c)The physical meaning of each Node (called a network node) is a feature map. The first feature map is the input of the cell, and the last feature map is the output of the cell. The lines between the feature maps represent the operation operators. Figure 7(a) defines the search space, the number of network nodes in each cell, the operator types in the search space, and three operators between two network nodes. Based on the operators given in the search space, each operator is assigned an initial weight value, and the weight of each operator is represented by the thickness of the line. In actual applications, the initial weight value of each operator between two nodes can be equal, that is, the possibility of each operator being selected is equal. Figure 7(b) shows that all operators in the cell update the weights corresponding to each operator through the gradient descent optimization strategy. The thickness of the line is used to represent the weight of each operator (indicating the possibility of being selected). The thicker the line, the greater the parameter weight, the greater the possibility of being selected. The thinner the line, the smaller the parameter weight, the smaller the possibility of being selected. After all parameters converge, operators with an ascending trend (ascending architecture parameters, AS) and operators with a descending trend (descending architecture parameters, DS) are determined. Figure 7(c) shows the cell architecture finally selected after training. The AS operator is retained, the DS operator is discarded, two operators are retained between Node-1 and Node-2, two operators are retained between Node-3 and Node-4, and one operator is retained between other nodes.

[0084] Exemplarily, in the process of searching for the down-sampling module and the up-sampling module, the objective function for searching the down-sampling module and the up-sampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2; where L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term; L edge is the image edge information loss term.

[0085] Here, L1 is specifically the 1-norm loss term between the network output and the given true value, and L2 is specifically the 2-norm loss term between the network output and the given true value.

[0086] In order to constrain computing energy consumption, the above objective function is used to make the searched network model more lightweight, while also taking into account accuracy and efficiency, making the network easier to deploy on mobile devices such as mobile phones, which can effectively reduce the computing energy consumption of mobile devices.

[0087] Exemplarily, in the process of searching for the downsampling module and the upsampling module, the performance measurement indicators used to search the downsampling module and the upsampling module based on the differentiable neural network structure include: Peak Signal Noise Ratio (PSNR) and Structural Similarity (SSIM). Here, PSNR and SSIM are used as image quality evaluation indicators. The larger the value, the smaller the image distortion and the better the network performance; otherwise, the worse the network performance.

[0088] Figure 8 Schematic diagram of the composition structure of the downsampling module in the embodiment of the present application, such as Figure 8 As shown in the figure, the searched downsampling module includes two input nodes (Input0 and Input1) and one output node (Output). D[XXX] represents the fusion of downsampling operator XXX, X is the identifier of each downsampling operator, and takes 0, 1, 2 or 3. N[XXX] represents the fusion of identity mapping operator XXX, X is the identifier of each identity mapping operator, and takes 0, 2 or 3. At the output end, the channel splicing is performed through the splicing node (Concat) to obtain the output feature map.

[0089] Fig. 9 Schematic diagram of the composition structure of the upsampling module in the embodiment of the present application. Fig. 9 As shown in the figure, the searched upsampling module includes two input nodes (Input0 and Input1) and one output node (Output). U[XXX] represents the fusion of upsampling operator XXX, X is the identifier of each upsampling operator, and takes 0 or 1. N[XXX] represents the fusion of identity mapping operator XXX, X is the identifier of each identity mapping operator, and takes 0, 2 or 3. At the output end, the channel splicing is performed through the splicing node (Concat) to obtain the output feature map.

[0090] Step 602: construct the image processing neural network using the down-sampling module, the up-sampling module and the shuffled attention module.

[0091] It should be noted that if Figure 3 As shown in the figure, the UNET network contains 4 layers, each of which contains the same downsampling module, upsampling module and shuffle attention module. The image processing neural network can be constructed by automatically searching for the minimum unit based on DARTS.

[0092] Exemplarily, in some embodiments, the image processing neural network further includes an input module and an output module; the input module is used to reduce the size of the input feature map by two times, and the output module is used to increase the size of the input feature map by two times, as well as upsampling by r times.

[0093] Step 603: Obtain a training data set.

[0094] Step 604: retrain the image processing neural network using the training data set until the loss function meets the convergence condition, thereby obtaining a trained image processing neural network.

[0095] That is to say, after the DARTS automatic search is completed, an image processing neural network is constructed based on all retained operators, and the image processing neural network is retrained using the training data to obtain the image processing neural network that can finally be applied.

[0096] During the search process, the above image processing neural network retains multiple valid operators and performs operator-level fusion, which can solve the problem of suboptimal model caused by unclear operator weight differences, reduce network calculations while ensuring network performance, and make the network easier to deploy on mobile devices such as mobile phones. While ensuring network performance, it can effectively reduce the computing energy consumption of mobile devices. The shuffled attention module is introduced into the UNET network to obtain a better-performing image processing neural network and improve the image super-resolution effect.

[0097] The present application also provides an image processing chip. Fig.10 As shown, the image processing chip 100 includes:

[0098] Processor 1001, used for acquiring an original image, and performing super-resolution reconstruction on the original image by using an image processing neural network to obtain a super-resolution image;

[0099] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules;

[0100] Wherein, the shuffled attention module is configured to: perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module.

[0101] In some embodiments, the shuffled attention module is further configured to extract channel attention feature map data and spatial attention feature map data of the feature map data output by the first downsampling module, and fuse the channel attention feature map data and the spatial attention feature map data and input them into the first upsampling module.

[0102] In some embodiments, the processor is further configured to:

[0103] Dividing the feature map data output by the first downsampling module into g groups of feature map data according to the dimensions;

[0104] Generate channel attention feature map data associated with the channel attention network and spatial attention feature map data associated with the spatial attention network based on each set of feature map data;

[0105] Concatenate the channel attention feature map data and the spatial attention feature map data of each set of feature map data to obtain g sets of concatenated feature map data; and

[0106] The channel shuffling operation and the operator fusion operation are performed on the g groups of spliced ​​feature map data to obtain fused feature map data.

[0107] In some embodiments, the image processing neural network further includes an input module and an output module; wherein the input module is used to reduce the size of the input feature map by a factor of two, and the output module is used to increase the size of the input feature map by a factor of two, as well as upsampling by a factor of r.

[0108] In some embodiments, the down-sampling module and the up-sampling module are searched based on a differentiable neural network structure;

[0109] The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node;

[0110] The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

[0111] In this way, during the search process, retaining multiple valid operators and performing operator-level fusion can solve the problem of suboptimal model caused by unclear differences in operator weights, and reduce the amount of network calculations while ensuring network performance.

[0112] In some embodiments, when searching the down-sampling module and the up-sampling module based on a micro-neural network structure, a first search space and a second search space are predefined; the first search space is searched based on a network parameter optimization strategy to obtain the down-sampling module; the second search space is searched based on a network parameter optimization strategy to obtain the up-sampling module;

[0113] Among them, the first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2.

[0114] In some embodiments, the network parameter optimization strategy includes: a first optimization strategy, selecting the first k operators with the largest weights; wherein k is an integer greater than 1; or a second optimization strategy, selecting operators whose weights have an increasing trend.

[0115] In some embodiments, the objective function for searching the down-sampling module and the up-sampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2;

[0116] Among them, L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term, L edge is the image edge information loss term.

[0117] In some embodiments, the performance metric indicators used to search the down-sampling module and the up-sampling module based on the differentiable neural network structure include: peak signal-to-noise ratio PSNR and structural similarity SSIM.

[0118] In some embodiments, the processor 1001 includes at least one of the following: a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural-network processing unit (NPU).

[0119] The image processing chip 100 may further include a memory 1002 for storing a computer program that can be run on the processor 1001 , and the processor 1001 is configured to perform the aforementioned super-resolution image reconstruction operation when running the computer program.

[0120] The above-mentioned memory can be a volatile memory (volatile memory), such as a random access memory (RAM); or a non-volatile memory (non-volatile memory), such as a read-only memory (ROM), a flash memory, a hard disk (HDD) or a solid-state drive (SSD); or a combination of the above-mentioned types of memory, and provide instructions and data to the processor.

[0121] Optionally, the image processing chip 100 may further include an input interface 1003. The input interface 1003 is used to communicate with other devices or chips, and specifically, information or data sent by other devices or chips may be obtained.

[0122] Optionally, the image processing chip 100 may further include an output interface 1004. The output interface 1004 communicates with other devices or chips, and specifically, may output information or data to other devices or chips.

[0123] Optionally, the image processing chip 100 may be applied to the electronic device in the embodiment of the present application.

[0124] Of course, in actual application, the various components in the chip are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0125] It should be noted that the above-mentioned image processing neural network in the embodiment of the present application can also be deployed on any other hardware computing unit for implementation.

[0126] The present application also provides a device for image processing, such as Fig.11 As shown, the device 110 includes:

[0127] An acquisition module 1101 is used to acquire an original image; and

[0128] A processing module 1102 is used to reconstruct the original image using an image processing neural network to obtain a super-resolution image;

[0129] The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules;

[0130] Wherein, the shuffled attention module is configured to: perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module.

[0131] In some embodiments, the shuffled attention module is further configured to extract channel attention feature map data and spatial attention feature map data of the feature map data output by the first downsampling module, and fuse the channel attention feature map data and the spatial attention feature map data and input them into the first upsampling module.

[0132] In some embodiments, the shuffle attention module includes: a first segmentation layer, a second segmentation layer, a channel attention network, a spatial attention network, a concatenation layer, and a shuffle fusion layer;

[0133] The first segmentation layer is configured to divide the feature map data output by the first downsampling module into g groups of feature map data according to dimensions;

[0134] The second segmentation layer is configured to generate channel attention feature map data associated with the channel attention network and spatial attention feature map data associated with the spatial attention network based on each set of feature map data;

[0135] The splicing layer is configured to splice the channel attention feature map data and the spatial attention feature map data of each set of feature map data to obtain g sets of spliced ​​feature map data;

[0136] The shuffle fusion layer is configured to perform the channel shuffle operation and the operator fusion operation on the g groups of spliced ​​feature map data to obtain fused feature map data.

[0137] In some embodiments, the image processing neural network further includes an input module and an output module; wherein the input module is used to reduce the size of the input feature map by a factor of two, and the output module is used to increase the size of the input feature map by a factor of two, as well as upsampling by a factor of r.

[0138] In some embodiments, the down-sampling module and the up-sampling module are searched based on a differentiable neural network structure;

[0139] The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node;

[0140] The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

[0141] In this way, during the search process, retaining multiple valid operators and performing operator-level fusion can solve the problem of suboptimal model caused by unclear differences in operator weights, and reduce the amount of network calculations while ensuring network performance.

[0142] In some embodiments, when searching the down-sampling module and the up-sampling module based on a micro-neural network structure, a first search space and a second search space are predefined; the first search space is searched based on a network parameter optimization strategy to obtain the down-sampling module; the second search space is searched based on a network parameter optimization strategy to obtain the up-sampling module;

[0143] Among them, the first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2.

[0144] In some embodiments, the network parameter optimization strategy includes: a first optimization strategy, selecting the first k operators with the largest weights; wherein k is an integer greater than 1; or a second optimization strategy, selecting operators whose weights have an increasing trend.

[0145] In some embodiments, the objective function for searching the down-sampling module and the up-sampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2;

[0146] Among them, L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term, L edge is the image edge information loss term.

[0147] In some embodiments, the performance metric indicators used to search the down-sampling module and the up-sampling module based on the differentiable neural network structure include: peak signal-to-noise ratio PSNR and structural similarity SSIM.

[0148] Exemplarily, at the head and tail of the UNET network, the input module is composed of a convolution operator with stride=2 and an activation function (Leaky ReLU). The output module is composed of a deconvolution operator with stride=2, an activation function (Leaky ReLU) and a pixel shuffle operator to achieve a two-fold decrease and a two-fold increase in the size of the input feature map. The PixelShuffle operator can map the feature map of size to a feature map of size rw×rh, thereby achieving r-fold upsampling of the image.

[0149] The above-mentioned device for image processing can be an electronic device or an image processing chip used in an electronic device. In the present application, the device can realize the functions of multiple units by software, hardware, or a combination of software and hardware, so that the device can execute the control method provided in any of the above-mentioned embodiments. The technical effects of each technical solution of the device can refer to the technical effects of the corresponding technical solution in the control method, and the present application will not elaborate on them one by one.

[0150] The present application also provides an electronic device, such as Fig.12 As shown, the electronic device 120 includes: a processor 1201 and a memory 1202 configured to store a computer program that can be run on the processor 1201,

[0151] The processor 1201 is configured to execute the method steps in the aforementioned embodiment when running a computer program.

[0152] In some embodiments, the processor 1201 includes at least one of the following: a graphics processing unit (GPU); a digital signal processor (DSP); an image signal processor (ISP); and a neural-network processing unit (NPU).

[0153] Of course, in practical applications, Fig.12 As shown, the various components in the electronic device are coupled together through a bus system. It can be understood that the bus system is used to realize the connection and communication between these components. In addition to the data bus, the bus system also includes a power bus, a control bus and a status signal bus.

[0154] The electronic devices described in this application may include, for example, mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), wearable devices, smart bracelets, cameras, etc.

[0155] In an exemplary embodiment, the embodiment of the present application further provides a computer-readable storage medium, such as a memory including a computer program, and the computer program can be executed by a processor of an electronic device to complete the steps of the aforementioned method.

[0156] It should be understood that the terms used in the present application are only for the purpose of describing specific embodiments, but are not intended to limit the present application. The singular forms of "a", "said" and "the" used in the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in this article refers to and includes any or all possible combinations of one or more associated listed items. The expressions "having", "may have", "include" and "include", or "may include" and "may include" in this application can be used to indicate the presence of corresponding features (e.g., elements such as numerical values, functions, operations or components), but the presence of additional features is not excluded.

[0157] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other and are not necessarily used to describe a specific order or sequence. For example, without departing from the scope of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0158] The technical solutions described in the embodiments of the present application can be combined arbitrarily without conflict.

[0159] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and equipment can be implemented in other ways. The embodiments described above are only schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0160] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0161] In addition, all functional units in the embodiments of the present application may be integrated into one processing unit, or each unit may be a separate unit, or two or more units may be integrated into one unit; the above-mentioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0162] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A method for image processing, characterized in that: The method comprises: Get the original image; Reconstructing the original image using an image processing neural network to obtain a super-resolution image; The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules; The shuffle attention module is configured to perform channel shuffle operations and operator fusion operations on the feature map data output by the first downsampling module. The method further comprises: Searching for the down-sampling module and the up-sampling module based on a differentiable neural network structure; Using the down-sampling module, the up-sampling module and the shuffled attention module, construct the image processing neural network; The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node; The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

2. The method according to claim 1, characterized in that The shuffled attention module is further configured to extract channel attention feature map data and spatial attention feature map data of the feature map data output by the first downsampling module, and fuse the channel attention feature map data and the spatial attention feature map data and input them into the first upsampling module.

3. The method according to claim 2, characterized in that The shuffle attention module includes: First segmentation layer, second segmentation layer, channel attention network, spatial attention network, concatenation layer and shuffle fusion layer; The first segmentation layer is configured to divide the feature map data output by the first downsampling module into g groups of feature map data according to the dimension; The second segmentation layer is configured to generate channel attention feature map data associated with the channel attention network and spatial attention feature map data associated with the spatial attention network based on each set of feature map data; The splicing layer is configured to splice the channel attention feature map data and the spatial attention feature map data of each set of feature map data to obtain g sets of spliced ​​feature map data; The shuffle fusion layer is configured to perform the channel shuffle operation and the operator fusion operation on the g groups of spliced ​​feature map data to obtain fused feature map data.

4. The method according to claim 1, characterized in that: The image processing neural network also includes an input module and an output module; The input module is used to reduce the size of the input feature map by two times, and the output module is used to increase the size of the input feature map by two times, and upsample by r times, where r is an integer greater than 1.

5. The method according to claim 1, characterized in that The searching for the down-sampling module and the up-sampling module based on the micro-neural network structure includes: pre-define a first search space and a second search space; Search the first search space based on a network parameter optimization strategy to obtain the downsampling module; Search the second search space based on a network parameter optimization strategy to obtain the upsampling module; The first search space includes: M network nodes, a downsampling operator and a first identity mapping operator; the second search space includes: N network nodes, an upsampling operator and a second identity mapping operator; M and N are both integers greater than or equal to 2; The network parameter optimization strategy includes one of the following: The first optimization strategy selects the first k operators with the largest weights; The second optimization strategy is to select operators whose weights have an upward trend; The third optimization strategy is to select an operator whose weight has an increasing trend from the first k operators with the largest weights; wherein k is an integer greater than 1.

6. The method according to claim 1, characterized in that The objective function for searching the down-sampling module and the up-sampling module based on the differentiable neural network structure is: argminL1+2·L ms-ssim +3·L edge +L2; Among them, L1 and L2 are norm loss terms, L ms-ssim is the image structure similarity loss term, L edge is the image edge information loss item; The performance metrics used to search the down-sampling module and the up-sampling module based on the differentiable neural network structure include: peak signal-to-noise ratio PSNR and structural similarity SSIM.

7. An image processing chip, characterized in that: include: A processor is used to obtain an original image, and perform super-resolution reconstruction on the original image using an image processing neural network to obtain a super-resolution image; The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules; Wherein, the shuffle attention module is configured to: perform a channel shuffle operation and an operator fusion operation on the feature map data output by the first downsampling module; The down-sampling module and the up-sampling module are searched based on a differentiable neural network structure; The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node; The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

8. The image processing chip according to claim 7, characterized in that: The processor is further configured to: Dividing the feature map data output by the first downsampling module into g groups of feature map data according to the dimensions; Generate channel attention feature map data of the channel attention network associated with the shuffled attention module and spatial attention feature map data of the spatial attention network associated with the shuffled attention module based on each set of feature map data; Concatenate the channel attention feature map data and the spatial attention feature map data of each set of feature map data to obtain g sets of concatenated feature map data; as well as The channel shuffling operation and the operator fusion operation are performed on the g groups of spliced ​​feature map data to obtain fused feature map data.

9. A device for image processing, characterized in that: include: An acquisition module, used for acquiring the original image; as well as A processing module, used to reconstruct the original image using an image processing neural network to obtain a super-resolution image; The image processing neural network is a UNET network obtained based on a differentiable neural network structure search, and the image processing neural network includes at least one down-sampling module and at least one up-sampling module corresponding to the at least one down-sampling module, wherein a first down-sampling module and a corresponding first up-sampling module are connected through a shuffled attention module, the first down-sampling module is one of the at least one down-sampling modules, and the first up-sampling module is one of the at least one up-sampling modules; Wherein, the shuffle attention module is configured to: perform a channel shuffle operation and an operator fusion operation on the feature map data output by the first downsampling module; The down-sampling module and the up-sampling module are searched based on a differentiable neural network structure; The downsampling module includes a first network node, at least two operators are included between the first network node and a previous network node, and the output of the previous network node is fused after passing through the at least two operators to obtain the input of the first network node; The upsampling module includes a second network node, and at least two operators are included between the second network node and the previous network node; the output of the previous network node is fused after passing through the at least two operators to obtain the input of the second network node.

10. An electronic device, characterized in that: include: a processor and a memory configured to store a computer program capable of being executed on said processor, Wherein, the processor is configured to execute the steps of the method according to any one of claims 1 to 6 when running the computer program.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images