Neural network channel number search method and apparatus

Abstract

A neural network channel number searching method and device can realize a differentiable search technology to search network channel numbers, reduce the calculation complexity of the network while ensuring the network performance. The method comprises the following steps: determining the output channel number N of a convolutional layer, wherein N is a positive integer (S301); dividing the feature tensor output by the convolutional layer into n sub-feature tensors, wherein the channel number of each sub-feature tensor is N / n, n is an integer that can be divided by N, and n is greater than or equal to 2 (S302); determining n sets of weighting coefficients, wherein each set of weighting coefficients comprises a plurality of weighting coefficients, and the plurality of weighting coefficients correspond to a plurality of sub-feature tensors in the n sub-feature tensors one by one (S303); determining the sub-feature tensor corresponding to the maximum value in each set of weighting coefficients (S304) to obtain the sub-feature tensors corresponding to the n maximum values; and re-determining the output channel number of the convolutional layer according to the sub-feature tensors corresponding to the n maximum values (S305).

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and more particularly, to a neural network channel number search method and device. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] With the rapid development of artificial intelligence technology, neural networks (for example, deep neural networks) have made great achievements in the processing and analysis of various media signals such as images, videos and speech in recent years. A good performance neural network often has a sophisticated network structure, which requires a human expert with superb skills and rich experience to spend a lot of effort to build. In order to better build a neural network, people have proposed a method of building a neural network through neural architecture search (NAS) to build a neural network, which searches for a neural network structure automatically to obtain a neural network structure with excellent performance.

[0004] Neural network structure search technology can be divided into different categories according to the search method. Differentiable search technology is one of the important technologies of NAS, which mainly includes three stages: constructing a differentiable neural network search space, searching for network structure, and decoding the search results to obtain the final network structure. When applying differentiable search technology to network structure search, the main search is currently for computing units, which can be an operation such as convolution, pooling, etc. or a block operation composed of multiple basic operations. When differentiable search technology searches for network structure, although different computing units can be searched, it does not support the search for the number of channels of a single convolution, and cannot meet the requirements when it is desired to search for a network with smaller computational complexity. SUMMARY

[0005] The present application provides a neural network channel number search method and device, which can realize the network channel number search problem of differentiable search technology, reduce the computational complexity of the network while ensuring the network performance.

[0006] In a first aspect, a neural network channel number searching method is provided, which comprises: determining an output channel number N of a convolution layer, N being a positive integer; dividing a feature tensor output by the convolution layer into n sub-feature tensors, each sub-feature tensor having a channel number of N / n, n being an integer that can be divided by N and n≥2; determining n sets of weighting coefficients, each set of weighting coefficients comprising a plurality of weighting coefficients, the plurality of weighting coefficients corresponding to a plurality of sub-feature tensors in the n sub-feature tensors one by one; determining a sub-feature tensor corresponding to a maximum value in each set of weighting coefficients in the n sets of weighting coefficients to obtain n sub-feature tensors corresponding to the n maximum values; and re-determining the output channel number of the convolution layer according to the n sub-feature tensors corresponding to the n maximum values.

[0007] In the application of the differentiable search technology for neural network structure search, currently, the main search object is a calculation unit, which can be an operation such as convolution, pooling, etc., or a block operation composed of multiple basic operations. Although the differentiable search technology can search different calculation units, it does not support the search of the channel number of a single convolution, and cannot meet the requirements when it is desired to search for a network with smaller calculation amount. The neural network channel number searching method provided in the embodiments of the present application can realize the search of the neural network channel number based on the differentiable search technology.

[0008] In combination with the first aspect, in a possible implementation manner, each set of weighting coefficients comprises n weighting coefficients, and the n weighting coefficients correspond to the n sub-feature tensors one by one.

[0009] In combination with the first aspect, in a possible implementation manner, each set of weighting coefficients comprises m weighting coefficients, and the m weighting coefficients correspond to m sub-feature tensors in the n sub-feature tensors one by one, where m is a positive integer smaller than n.

[0010] The embodiments of the present application provide two possible implementation manners to determine the sub-feature tensor corresponding to the maximum value in each set of weighting coefficients in the n sets of weighting coefficients, i.e., the sub-feature tensor corresponding to each maximum value can be determined from the n sub-feature tensors or from part of the n sub-feature tensors.

[0011] In combination with the first aspect, in a possible implementation manner, before determining the sub-feature tensor corresponding to the maximum value in each set of weighting coefficients in the n sets of weighting coefficients, the method further comprises: generating n candidate feature tensors according to the n sets of weighting coefficients and a plurality of sub-feature tensors in the n sub-feature tensors, wherein one set of weighting coefficients corresponds to one candidate feature tensor.

[0012] With reference to the first aspect, in a possible implementation manner, the sub-feature tensor corresponding to the maximum value in each group of the n groups of weighting coefficients is determined to obtain n sub-feature tensors corresponding to the maximum values.

[0013] In addition to the method provided in the first aspect, the embodiment of the present application further provides a method for determining the sub-feature tensor corresponding to the maximum value. That is, each candidate feature tensor can be calculated according to each group of weighting coefficients and a plurality of sub-feature tensors in the n sub-feature tensors, wherein the plurality of sub-feature tensors can be the n sub-feature tensors, or can be part of the n sub-feature tensors, and then the sub-feature tensor with the maximum weight in each candidate feature tensor is determined as the sub-feature tensor corresponding to the maximum value.

[0014] With reference to the first aspect, in a possible implementation manner, the output channel number of the convolutional layer is re-determined according to the n sub-feature tensors corresponding to the maximum values, comprising: determining the number k of sub-feature tensors corresponding to the maximum values in the n sub-feature tensors corresponding to the maximum values, k being a positive integer less than or equal to n; and the re-determined output channel number of the convolutional layer being kN / n.

[0015] The output channel number of the convolutional layer is re-determined according to the number of sub-feature tensors corresponding to the maximum values in the n sub-feature tensors corresponding to the maximum values, and the re-determined channel number is k / n of the original, so that the channel number of the neural network can be compressed, thereby reducing the computational complexity of the neural network.

[0016] The second aspect provides an image processing method, comprising: obtaining a to-be-processed image; and classifying the to-be-processed image according to a target neural network to obtain a classification result of the to-be-processed image; wherein the determination of the channel number of the target neural network comprises: determining an output channel number N of a convolutional layer, N being a positive integer; dividing a feature tensor output by the convolutional layer into n sub-feature tensors, each sub-feature tensor having a channel number of N / n, n being an integer that can be divided by N and n≥2; determining n groups of weighting coefficients, each group of weighting coefficients comprising a plurality of weighting coefficients, the plurality of weighting coefficients corresponding to a plurality of sub-feature tensors in the n sub-feature tensors; determining a sub-feature tensor corresponding to a maximum value in each group of the n groups of weighting coefficients to obtain n sub-feature tensors corresponding to the maximum values; and re-determining the output channel number of the convolutional layer according to the n sub-feature tensors corresponding to the maximum values.

[0017] The neural network searched by the neural network channel number searching method provided in the embodiment of the present application is used for image processing, and the overall computational complexity of the neural network is reduced compared with the neural network without channel number compression.

[0018] With reference to the second aspect, in a possible implementation manner, each group of the weighting coefficients includes n weighting coefficients, and the n weighting coefficients correspond to the n sub-feature tensors one by one.

[0019] With reference to the second aspect, in a possible implementation manner, each group of the weighting coefficients includes m weighting coefficients, and the m weighting coefficients correspond to m sub-feature tensors of the n sub-feature tensors one by one, where m is a positive integer less than n.

[0020] With reference to the second aspect, in a possible implementation manner, before determining the sub-feature tensor corresponding to the maximum value in each group of the n groups of the weighting coefficients, the method further includes: generating n candidate feature tensors according to the n groups of the weighting coefficients and a plurality of sub-feature tensors of the n sub-feature tensors, where one group of the weighting coefficients corresponds to one candidate feature tensor.

[0021] With reference to the second aspect, in a possible implementation manner, the determining of the sub-feature tensor corresponding to the maximum value in each group of the n groups of the weighting coefficients to obtain the sub-feature tensors corresponding to the n maximum values further includes: determining a sub-feature tensor with the maximum weight from the plurality of sub-feature tensors generating each candidate feature tensor to obtain n sub-feature tensors with the maximum weight.

[0022] With reference to the second aspect, in a possible implementation manner, the re-determining of the output channel number of the convolutional layer according to the sub-feature tensors corresponding to the n maximum values includes: determining a number k of sub-feature tensors that are different from each other in the sub-feature tensors corresponding to the n maximum values, where k is a positive integer less than or equal to n; and the re-determined output channel number of the convolutional layer is kN / n.

[0023] According to a third aspect, a neural network channel number searching apparatus is provided, including: a memory configured to store a program; and a processor configured to execute the program stored in the memory, when the program stored in the memory is executed, the processor is configured to perform the following process: determining an output channel number N of a convolutional layer, where N is a positive integer; dividing a feature tensor output by the convolutional layer into n sub-feature tensors, where a channel number of each sub-feature tensor is N / n, n is an integer that can be divided by N, and n is greater than or equal to 2; determining n groups of weighting coefficients, where each group of the weighting coefficients includes a plurality of weighting coefficients, and the plurality of weighting coefficients correspond to a plurality of sub-feature tensors of the n sub-feature tensors one by one; determining a sub-feature tensor corresponding to a maximum value in each group of the n groups of the weighting coefficients to obtain n sub-feature tensors corresponding to the n maximum values; and re-determining an output channel number of the convolutional layer according to the sub-feature tensors corresponding to the n maximum values.

[0024] In a fourth aspect, an image processing apparatus is provided, including: a memory configured to store a program; and a processor configured to execute the program stored in the memory, and when the program stored in the memory is executed, the processor is configured to perform the following process: obtaining a to-be-processed image; and classifying the to-be-processed image according to a target neural network to obtain a classification result of the to-be-processed image, wherein determining a number of channels of the target neural network includes: determining an output channel number N of a convolutional layer, N being a positive integer; dividing a feature tensor output by the convolutional layer into n sub-feature tensors, a number of channels of each sub-feature tensor being N / n, n being an integer that can be divided by N and n≥2; determining n sets of weighting coefficients, each set of weighting coefficients including a plurality of weighting coefficients, the plurality of weighting coefficients corresponding to a plurality of sub-feature tensors in the n sub-feature tensors one by one; determining a sub-feature tensor corresponding to a maximum value in each set of weighting coefficients in the n sets of weighting coefficients to obtain n sub-feature tensors corresponding to the n maximum values; and re-determining the output channel number of the convolutional layer according to the n sub-feature tensors corresponding to the n maximum values.

[0025] In a fifth aspect, a computer readable storage medium is provided, the computer readable medium storing program code for execution by an apparatus, the program code including code for performing the method in any of the implementations of the first aspect to the second aspect.

[0026] In a sixth aspect, a computer program product including instructions, which when executed on a computer, cause the computer to perform the method in any of the implementations of the first aspect to the second aspect.

[0027] In a seventh aspect, a chip is provided, the chip including a processor and a data interface, the processor reading instructions stored on a memory through the data interface to perform the method in any of the implementations of the first aspect to the second aspect.

[0028] Optionally, as one possible implementation, the chip can further include a memory, the memory storing instructions, and the processor being configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to perform the method in any of the implementations of the first aspect to the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a structural schematic diagram of a convolutional neural network provided by an embodiment of the present application;

[0030] Figure 2 is a schematic block diagram of a differentiable neural network structure search method provided by an embodiment of the present application;

[0031] Figure 3is a schematic flow chart of a neural network channel number search method provided by an embodiment of the present application;

[0032] Figure 4 is a schematic block diagram of a neural network channel number search method provided by an embodiment of the present application;

[0033] Figure 5 is a schematic block diagram of another neural network channel number search method provided by an embodiment of the present application;

[0034] Figure 6 is a schematic diagram of a super-resolution neural network structure provided by an embodiment of the present application;

[0035] Figure 7 is a schematic flow chart of an image processing method provided by an embodiment of the present application;

[0036] Figure 8 is a schematic hardware structure diagram of a neural network channel number search device provided by an embodiment of the present application;

[0037] Figure 9 is a schematic hardware structure diagram of an image processing device provided by an embodiment of the present application;

[0038] Figure 10 is a hardware structure diagram of a neural network training device according to an embodiment of the present application;

[0039] Figure 11 is a schematic structural block diagram of a neural network channel number search device provided by an embodiment of the present application;

[0040] Figure 12 is a schematic structural block diagram of an image processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0041] The technical solutions in the present application will be described below with reference to the accompanying drawings.

[0042] The neural network obtained by the neural network channel number search method provided by an embodiment of the present application can be a convolutional neural network (CNN), a deep convolutional neural network (DCNN), a recurrent neural network (RNN), etc. Since CNN is a very common neural network, the structure of CNN will be introduced below. Figure 1

[0043] The structure of the neural network actually used by the image processing method provided by an embodiment of the present application can be as shown in Figure 1 the accompanying drawings.​Figure 1 In this example, the Convolutional Neural Network (CNN) 100 may include an input layer 110, convolutional / pooling layers 120 (where the pooling layers are optional), and a neural network layer 130. The input layer 110 acquires the image to be processed and then passes the acquired image to the convolutional / pooling layers 120 and the subsequent neural network layer 130 for processing to obtain the image processing result. The following section... Figure 1 This section provides a detailed introduction to the internal layer structure of CNN 100.

[0044] Convolutional / pooling layers 120:

[0045] Convolutional layers:

[0046] like Figure 1 The convolutional / pooling layer 120 shown may include layers as in Examples 121-126. For instance, in one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0047] The following section will use convolutional layer 121 as an example to introduce the internal working principle of a convolutional layer.

[0048] The convolution layer 121 can include a plurality of convolution operators, also known as kernels, which function as filters in image processing to extract specific information from an input image matrix. The convolution operator can be essentially a weight matrix, which is usually predefined. In the process of convolution operation on the image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as that of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix will produce a single-depth convolution output, but most cases do not use a single weight matrix, but apply a plurality of weight matrices of the same size (rows x columns), i.e. a plurality of homogeneous matrices. The output of each weight matrix is stacked to form the depth dimension of the convolution image, where the dimension can be understood as determined by the "plurality" described above. Different weight matrices can be used to extract different features in the image, such as a weight matrix for extracting image edge information, another weight matrix for extracting specific colors of the image, and another weight matrix for blurring unwanted noise in the image, etc. The plurality of weight matrices are of the same size (rows x columns), and the size of the convolution feature map extracted by the plurality of weight matrices of the same size is also the same. The plurality of convolution feature maps of the same size are combined to form the output of the convolution operation.

[0049] The weight values in these weight matrices need to be obtained through a large amount of training in actual applications. Each weight matrix formed by the weight values obtained through training can extract information from the input image, thereby enabling the convolutional neural network 100 to make correct predictions.

[0050] When the convolutional neural network 100 has multiple convolution layers, the initial convolution layer (e.g. 121) often extracts more general features, which can also be referred to as low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolution layers (e.g. 126) become more and more complex, such as high-level semantic features, etc. The higher the semantic features, the more suitable they are for the problem to be solved.

[0051] Pooling layer:

[0052] Since it is often necessary to reduce the number of training parameters, a pooling layer is often periodically introduced after the convolution layer, i.e. as Figure 1Each of the layers 121-126 of the example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the only purpose of a pooling layer is to reduce the spatial size of an image. A pooling layer can include an average pooling operator and / or a max pooling operator to sample the input image to produce a smaller image. An average pooling operator can calculate the average value of the pixel values in a certain range to produce an average value as the result of average pooling. A max pooling operator can take the maximum value of the pixels in a certain range as the result of max pooling. In addition, just like the size of the weight matrix in a convolutional layer should be related to the size of the image, the operator in a pooling layer should also be related to the size of the image. The size of the image output by a pooling layer can be smaller than the size of the image input to the pooling layer, and each pixel point in the image output by the pooling layer represents the average value or the maximum value of the corresponding sub-region of the image input to the pooling layer.

[0053] The neural network layer 130:

[0054] After processing by the convolutional / pooling layer 120, the convolutional neural network 100 is still not sufficient to output the required output information. As described above, the convolutional / pooling layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 100 needs to use the neural network layer 130 to generate an output of the number of required classes or a group of outputs. Therefore, the neural network layer 130 can include multiple hidden layers (such as 131, 132, and 13n shown) and an output layer 140, and the parameters contained in the multiple hidden layers can be pre-trained according to the related training data of a specific task type, for example, the task type can include image recognition, image classification, image super-resolution reconstruction, etc. Figure 1

[0055] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation (such as the propagation from 110 to 140) of the entire convolutional neural network 100 is completed, the back propagation (such as the propagation from 140 to 110) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the result output by the output layer of the convolutional neural network 100 and the ideal result. Figure 1 Figure 1

[0056] Figure 1 ​​​The neural network shown can be obtained by a neural network structure search method. There are various categories of neural network structure search methods, and the differentiable search technique is one of the important techniques for neural network structure search, which mainly includes three stages: constructing a differentiable neural network search space, performing network structure search, and decoding the search results to obtain the final network structure. The following will be introduced in combination with Figure 2 The differentiable neural network structure search technique is briefly introduced.

[0057] The first step is to construct a differentiable neural network search space. As shown in Figure 2 The candidate network computing units 1, 2, 3, 4, and 5 are deployed in a network, which is constructed by weighted summation. The weighting coefficients are obtained by the gunbel_softmax conversion function, which can convert the weighting coefficients into a vector between 0 and 1 and add up to 1. The output distribution is controlled by the temperature parameter. When the temperature is very low, the output tends to be an average distribution. When the temperature is very high, the output tends to be a one-hot distribution, i.e., only one element tends to 1 and the other elements tend to 0. In Figure 2 a1, a2, a3, a4, and a5 are weighting coefficients, where a1 is the weighting coefficient of computing unit 1, a2 is the weighting coefficient of computing unit 2, and so on.

[0058] The second step is network structure search. For the network constructed in the first step, the network parameters and weighting coefficients of the network computing units are alternately trained, where the weighting coefficients represent the structure parameters of the network structure. Specifically, the input data is transmitted into computing units 1 to 5 for network parameter training of the computing units. Then the data processed by computing units 1 to 5 are multiplied by weighting coefficients a1 to a5 and added to obtain output 1. The data of output 1 are transmitted into computing units 1' to 5' for network parameter training of the computing units. Then the data processed by computing units 1' to 5' are multiplied by weighting coefficients b1 to b5 and added to obtain output 2.

[0059] The third step is to decode the search results. When the network training in the second step is completed, according to the final weighting coefficients, for each weighted summation term, the computing unit with the largest weighting coefficient is retained and the other computing units are deleted to obtain the final network structure as the search result. Specifically, according to the largest weighting coefficient among weighting coefficients a1 to a5, the corresponding computing unit is retained and the other computing units are deleted; according to the largest weighting coefficient among weighting coefficients b1 to b5, the corresponding computing unit is retained and the other computing units are deleted.

[0060] In the neural network structure search using the differentiable search technology, currently, the search is mainly performed on the calculation unit, and the calculation unit can be an operation such as convolution, pooling, etc., or a block operation composed of multiple basic operations. Although the differentiable search technology can search different calculation units, it does not support the search of the channel number of a single convolution, and cannot meet the requirement when it is desired to search a network with smaller calculation amount. The neural network channel number search method provided in the embodiments of the present application can search the neural network channel number based on the differentiable search technology.

[0061] Figure 3 A schematic flowchart of the neural network channel number search method provided in the present application is shown. Figure 3 The method shown can be performed by a neural network structure search device, which can be a computer, a server, a cloud device, or the like, and has sufficient computing capability to search the neural network structure. Figure 3 The method shown includes steps 301 to 305, which are described in detail below.

[0062] S301, determine the output channel number N of the convolution layer, N being a positive integer.

[0063] The output channel number N of the convolution layer can be the maximum output channel number of the convolution layer, and the maximum output channel number of the convolution layer can be a value set according to a specific embodiment.

[0064] S302, divide the feature tensor output by the convolution layer into n sub-feature tensors, and the channel number of each sub-feature tensor is N / n, n being an integer that can be divided by N and n≥2.

[0065] The division of the feature tensor output by the convolution layer is a division in the channel dimension, for example, a picture is a three-dimensional tensor, and the length and width are one dimension, and the third dimension is the channel number. Dividing the feature tensor in the channel dimension can make each sub-feature tensor evenly divide the channel number of the convolution layer.

[0066] S303, determine n sets of weighting coefficients, and each set of weighting coefficients includes multiple weighting coefficients, and the multiple weighting coefficients correspond one-to-one to multiple sub-feature tensors in the n sub-feature tensors.

[0067] Optionally, each group of weighting coefficients includes n weighting coefficients, and the n weighting coefficients correspond to the n sub-feature tensors one by one. For example, 4 groups of weighting coefficients are determined, and each group of weighting coefficients includes 4 weighting coefficients a1, a2, a3, and a4, which correspond to 4 sub-feature tensors T1, T2, T3, and T4 one by one, that is, a1 corresponds to T1, a2 corresponds to T2, a3 corresponds to T3, and a4 corresponds to T4. It should be understood that each group of weighting coefficients can be different from each other.

[0068] Optionally, each group of weighting coefficients includes m weighting coefficients, and the m weighting coefficients correspond to m sub-feature tensors in the n sub-feature tensors one by one, where m is a positive integer less than n. For example, 4 groups of weighting coefficients are determined, and each group of weighting coefficients includes 2 weighting coefficients a1 and a2, which correspond to any two of 4 sub-feature tensors T1, T2, T3, and T4 one by one, for example, a1 corresponds to T1 and a2 corresponds to T2 in the first group, a1 corresponds to T1 and a2 corresponds to T3 in the second group, and so on. It should be understood that each group of weighting coefficients can be different from each other.

[0069] The embodiments of the present application provide two possible implementation manners to determine the sub-feature tensor corresponding to the maximum value in each group of weighting coefficients in the n groups of weighting coefficients, that is, the sub-feature tensor corresponding to each maximum value can be determined from the n sub-feature tensors or from part of the n sub-feature tensors.

[0070] S304, determining the sub-feature tensor corresponding to the maximum value in each group of weighting coefficients in the n groups of weighting coefficients to obtain the sub-feature tensors corresponding to the n maximum values.

[0071] In combination with the example in S303, for example, the maximum weighting coefficient in the first group is a1, and the sub-feature tensor corresponding to the maximum value is T1; the maximum weighting coefficient in the second group is also a1, and the sub-feature tensor corresponding to the maximum value is also T1; the maximum weighting coefficient in the third group is a2, and the sub-feature tensor corresponding to the maximum value is T2; the maximum weighting coefficient in the fourth group is a3, and the sub-feature tensor corresponding to the maximum value is T3. The sub-feature tensors corresponding to the 4 maximum values obtained are T1, T1, T2, and T3, respectively.

[0072] Optionally, before determining the sub-feature tensors corresponding to the maximum values in each of the n groups of weighting coefficients, n candidate feature tensors can be generated according to the n groups of weighting coefficients and a plurality of sub-feature tensors in the n sub-feature tensors, wherein one group of weighting coefficients corresponds to one candidate feature tensor. Then, the sub-feature tensor with the maximum weight in the plurality of sub-feature tensors for generating each candidate feature tensor is determined to obtain n sub-feature tensors with the maximum weight. In combination with the example in S303, one group of weighting coefficients is taken as an example, for example, one candidate feature tensor TC1 is generated according to the four weighting coefficients a1, a2, a3, a4 in the group and the four sub-feature tensors T1, T2, T3, T4, that is, TC1=a1xT1+a2xT2+a3xT3+a4xT4, and then the sub-feature tensor with the maximum weight in the generation of TC1, for example, T1, can be determined. Thus, n sub-feature tensors with the maximum weight can be obtained.

[0073] S305, the output channel number of the convolutional layer is re-determined according to the sub-feature tensors corresponding to the n maximum values.

[0074] Specifically, the number k of sub-feature tensors that are different from each other in the n sub-feature tensors corresponding to the n maximum values can be determined, k is a positive integer less than or equal to n, and then the output channel number of the re-determined convolutional layer is kN / n.

[0075] In combination with the example in S304, it has been obtained that the sub-feature tensors corresponding to the four maximum values are T1, T1, T2, and T3, wherein the sub-feature tensors that are different from each other are T1, T2, and T3, and the number is 3. Thus, the output channel number of the re-determined convolutional layer can be 3N / 4. Thus, the compression of the neural network channel number can be realized, so as to reduce the calculation complexity of the neural network.

[0076] In order to better understand the neural network channel number searching method provided by the embodiments of the present application, the following will be described in combination with Figure 4 The overall process of the neural network channel number searching method of the embodiments of the present application will be introduced. Taking the convolutional layer 1 to the convolutional layer 2 as an example, the search space method for searching the convolutional channel number constructed by the embodiments of the present application is as follows.

[0077] First, the maximum output channel number N of the convolutional layer is determined, which can be a value set according to specific embodiments. The maximum value of the search channel number is determined according to the maximum output channel number N of the convolutional layer.

[0078] The feature tensor T output by the convolutional layer 1 is divided into four sub-feature tensors T0, T1, T2, and T3 in the channel dimension as shown in the following formula (1): Figure 4 Thus, the channel number of each sub-feature tensor is N / 4.

[0079] Figure 4The four sub-feature tensors are weighted and summed to generate a candidate feature tensor. This process is repeated four times to generate four candidate feature tensors TC0, TC1, TC2, and TC3. The weighting coefficients of the weighted summation are obtained by processing the sum using the gumbel_softmax transformation function. The definition of the gumbel_softmax transformation function is as follows:

[0080]

[0081] Where g is a randomly generated variable, τ is a set value, π is the input, and y is the output, which is the calculated weighting coefficient. For example, with a set of inputs π01, π02, π03, and π04, a set of corresponding weighting coefficients a01, a02, a03, and a04 can be calculated, where a01, a02, a03, and a04 are all vectors between [0, 1] that add up to 1. Similarly, four sets of weighting coefficients can be obtained.

[0082] For the four sets of inputs, the weighting coefficients after processing by the gumbel_softmax transformation function are shown in Table 1.

[0083] Table 1

[0084] a00 a10 a20 a30 a01 a11 a21 a31 a02 a12 a21 a32 a03 a13 a23 a33

[0085] Based on the four sets of weighted coefficients and four sub-feature tensors in Table 1, four candidate feature tensors TC0, TC1, TC2, and TC3 can be obtained. Among them, the candidate feature tensors TC0 is a00×T0+a01×T1+a02×T2+a03×T3, TC1 is a10×T0+a11×T1+a12×T2+a13×T3, TC2 is a20×T0+a21×T1+a22×T2+a23×T3, and TC3 is a30×T0+a31×T1+a32×T2+a33×T3.

[0086] The four candidate feature tensors TC0, TC1, TC2 and TC3 are concatenated into a single feature tensor Tout, which is used as the output of convolutional layer 1. The number of channels remains N.

[0087] The feature tensor Tout is input into the next convolutional layer 2.

[0088] After the search is complete, the sub-feature tensor that contributes the most to each candidate feature tensor can be obtained. For example, for TC0, a00>a01>a02>a03, so the sub-feature tensor that contributes the most to TC0 is T0; for TC1, a10>a11>a12>a13, so the sub-feature tensor that contributes the most to TC1 is T0; for TC2, a23>a20>a21>a22, so the sub-feature tensor that contributes the most to TC2 is T3; for TC3, a32>a30>a31>a33, so the sub-feature tensor that contributes the most to TC3 is T2. Since the sub-feature tensor that contributes the most to both TC0 and TC1 is T0, only T0, T2, and T3 need to be retained from the four sub-feature tensors T0, T1, T2, and T3. Therefore, the number of channels only needs to be 3N / 4, which can achieve channel compression.

[0089] The network structure obtained by this search only requires 3 / 4 of the original number of channels in convolutional layer 1. This reduces the number of channels and the overall computational complexity of the network.

[0090] Optionally, Figure 5 This demonstrates another way to generate candidate feature tensors. For example... Figure 5 As shown, the sub-feature tensor T0 is directly used as the candidate feature tensor TC0. The sub-feature tensors T0 and T1 are weighted and summed to generate the candidate feature tensor TC1 as a10×T0+a11×T1. The sub-feature tensors T0 and T2 are weighted and summed to generate the candidate feature tensor TC2 as a20×T0+a21×T2. The sub-feature tensors T0 and T3 are weighted and summed to generate the candidate feature tensor TC3 as a30×T0+a31×T3. The sub-feature tensor that contributes the most to TC0 is obviously T0. For TC1, a10>a11, therefore the sub-feature tensor that contributes the most to TC1 is T0. For TC2, a20>a21, therefore the sub-feature tensor that contributes the most to TC2 is T0. For TC3, a31>a30, therefore the sub-feature tensor that contributes the most to TC3 is T3. Since the sub-feature tensor that contributes the most in TC0, TC1, and TC2 is T0, only T0 and T3 need to be retained among the four sub-feature tensors T0, T1, T2, and T3. Therefore, the number of channels only needs to be N / 2, which can achieve channel compression.

[0091] Figure 5 The combination of candidate feature tensors shown is beneficial for searching for network structures with fewer channels.

[0092] Figure 6The super-resolution neural network structure searched by the neural network channel number searching method provided in the embodiments of the present application is shown in the schematic diagram. The performance of the super-resolution neural network structure searched by the neural network channel number searching method provided in the embodiments of the present application is comparable to that of the super-resolution neural network structure not searched by the neural network channel number searching method provided in the embodiments of the present application. Among them, the channel number of the convolution layer 0 of the super-resolution neural network structure searched by the neural network channel number searching method provided in the embodiments of the present application is reduced by 50%, the channel number of the convolution layer 1 is reduced by 50%, the channel number of the convolution layer 2 is reduced by 0%, the channel number of the convolution layer 3 is reduced by 0%, and the channel number of the convolution layer 4 is reduced by 50%. After the channel number is compressed, the overall network calculation complexity is reduced by 37%.

[0093] The effect of the neural network structure searched by the method provided in the embodiments of the present application on the image after processing is comparable to the subjective effect of the original image, and is better than the effect of processing the image using the difference method. In terms of details, the details of the image processed by the neural network structure searched by the method provided in the embodiments of the present application are clearer than those of the original image.

[0094] Figure 7 is a schematic flowchart of the image processing method of the embodiments of the present application. It should be understood that the related content limitations, explanations and extensions of the method shown in the above are also applicable to the method shown in Figure 3 , and the repeated description will be appropriately omitted in the following introduction of the method shown in Figure 7 . The method shown in Figure 7 can be applied to a terminal device, which includes: Figure 7

[0095] S701, an image to be processed is acquired.

[0096] S702, the image to be processed is classified according to a target neural network to obtain a classification result of the image to be processed.

[0097] The determination of the channel number of the target neural network includes: determining an output channel number N of a convolution layer, N being a positive integer; a feature tensor output by the convolution layer is divided into n sub-feature tensors, the channel number of each sub-feature tensor being N / n, n being an integer that can be divided by N and n≥2; n sets of weighting coefficients are determined, each set of weighting coefficients including multiple weighting coefficients, the multiple weighting coefficients corresponding to multiple sub-feature tensors in the n sub-feature tensors one by one; a sub-feature tensor corresponding to a maximum value in each set of weighting coefficients in the n sets of weighting coefficients is determined to obtain n sub-feature tensors corresponding to the maximum values; and the output channel number of the convolution layer is re-determined according to the n sub-feature tensors corresponding to the maximum values.

[0098] Figure 8 is a hardware structure schematic diagram of the neural network channel number searching device provided in the embodiments of the present application. Figure 8 ​The illustrated neural network channel number searching apparatus 800 (which can be specifically a computer device) comprises a memory 801, a processor 802, a communication interface 803 and a bus 804. The memory 801, the processor 802 and the communication interface 803 are in communication connection with each other through the bus 804.

[0099] The memory 801 can be a read only memory (ROM), a static storage device, a dynamic storage device or a random access memory (RAM). The memory 801 can store a program, and when the program stored in the memory 801 is executed by the processor 802, the processor 802 is configured to execute each step of the neural network channel number searching method of the embodiments of the present application.

[0100] The processor 802 can be a general central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), a graphics processing unit (GPU) or one or more integrated circuits, configured to execute relevant programs to implement the neural network channel number searching method of the embodiments of the present application.

[0101] The processor 802 can also be an integrated circuit chip with signal processing capability. In the implementation process, each step of the neural network channel number searching method of the present application can be completed by the integrated logic circuit of hardware or the instruction in the form of software in the processor 802.

[0102] The processor 802 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor to execute, or executed by a combination of hardware and software modules in the code processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the memory 801, and the processor 802 reads the information in the memory 801, and combines the hardware to complete the functions required by the units included in the neural network channel number search device, or executes the neural network channel number search method of the method embodiments of the present application.

[0103] The communication interface 803 uses a transceiver such as but not limited to a transceiver to realize the communication between the device 800 and other devices or communication networks. For example, the information of the target neural network to be determined and the training data required in the process of determining the target neural network can be obtained through the communication interface 803.

[0104] The bus 804 can include a path for transmitting information between various components (e.g., the memory 801, the processor 802, the communication interface 803) of the device 800.

[0105] Figure 9 The figure shows the hardware structure of the image processing device of the embodiments of the present application. Figure 9 The image processing device 900 shown includes a memory 901, a processor 902, a communication interface 903 and a bus 904. Among them, the memory 901, the processor 902 and the communication interface 903 are communicatively connected with each other through the bus 904.

[0106] The memory 901 can be a ROM, a static storage device and a RAM. The memory 901 can store programs, and when the programs stored in the memory 901 are executed by the processor 902, the processor 902 and the communication interface 903 are used to execute the various steps of the image processing method of the embodiments of the present application.

[0107] The processor 902 can be a general-purpose, CPU, microprocessor, ASIC, GPU, or one or more integrated circuits that perform the functions of the units in the image processing apparatus according to the embodiments of the present application, or perform the image processing method according to the method embodiments of the present application.

[0108] The processor 902 can also be an integrated circuit chip with a processing capability of signals. In the implementation process, each step of the image processing method according to the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor 902 or the instructions in the form of software.

[0109] The processor 902 described above can also be a general-purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps, and logic block diagrams in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware coding processor for execution, or a combination of hardware and software modules in the coding processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the storage 901, and the processor 902 reads the information in the storage 901, and combines the hardware to complete the functions required to be executed by the units included in the image processing apparatus according to the embodiments of the present application, or to execute the image processing method according to the method embodiments of the present application.

[0110] The communication interface 903 uses a transceiver such as but not limited to a transceiver to realize the communication between the apparatus 900 and other devices or communication networks. For example, the image to be processed can be obtained through the communication interface 903.

[0111] The bus 904 can include a path for transmitting information between the various components (for example, the storage 901, the processor 902, the communication interface 903) of the apparatus 900.

[0112] Figure 10 is a hardware structure schematic diagram of the neural network training apparatus according to the embodiments of the present application. Similar to the apparatuses 800 and 900 described above, Figure 10 The neural network training apparatus 1000 shown includes a storage 1001, a processor 1002, a communication interface 1003, and a bus 1004. Among them, the storage 1001, the processor 1002, and the communication interface 1003 are communicatively connected with each other through the bus 1004.

[0113] In the process of obtaining the image to be processed, the communication interface 1003 can obtain the image to be processed through the communication network. Figure 8After the neural network is searched by the neural network channel number searching apparatus shown in the figure, the neural network can be obtained by Figure 10 The neural network training apparatus 1000 shown in the figure trains the neural network, and the trained neural network can be used to perform the image processing method of the embodiments of the present application.

[0114] Specifically, Figure 10 The apparatus shown in the figure can obtain training data and a neural network to be trained from the outside through the communication interface 1003, and then the processor can train the neural network to be trained according to the training data.

[0115] It should be noted that although the apparatus 800, the apparatus 900 and the apparatus 1000 shown above only show the memory, the processor and the communication interface, in the specific implementation process, those skilled in the art should understand that the apparatus 800, the apparatus 900 and the apparatus 1000 can also include other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the apparatus 800, the apparatus 900 and the apparatus 1000 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the apparatus 800, the apparatus 900 and the apparatus 1000 can also only include the devices necessary for realizing the embodiments of the present application, and do not necessarily include all the devices shown in the above Figure 8 、 Figure 9 and Figure 10 .

[0116] Figure 11 is a schematic structural block diagram of a neural network channel number searching apparatus provided by the embodiments of the present application, wherein the neural network channel number searching apparatus 1100 comprises:

[0117] A first determining unit 1101 is configured to determine an output channel number N of a convolutional layer, wherein N is a positive integer;

[0118] A dividing unit 1102 is configured to divide a feature tensor output by the convolutional layer into n sub-feature tensors, wherein a channel number of each of the n sub-feature tensors is N / n, n is an integer that can be divided by N, and n≥2;

[0119] A second determining unit 1103 is configured to determine n sets of weighting coefficients, wherein each set of the n sets of weighting coefficients comprises a plurality of weighting coefficients, and the plurality of weighting coefficients one-to-one correspond to a plurality of sub-feature tensors in the n sub-feature tensors;

[0120] A third determining unit 1104 is configured to determine a sub-feature tensor corresponding to a maximum value in each set of the n sets of weighting coefficients, to obtain n sub-feature tensors corresponding to the maximum values;

[0121] The updating unit 1105 is configured to update the output channel number of the convolutional layer according to the sub-feature tensors corresponding to the n maximum values.

[0122] In an implementation, each set of weighting coefficients includes n weighting coefficients, and the n weighting coefficients correspond to the n sub-feature tensors.

[0123] In an implementation, each set of weighting coefficients includes m weighting coefficients, and the m weighting coefficients correspond to m sub-feature tensors in the n sub-feature tensors, where m is a positive integer less than n.

[0124] In an implementation, the third determining unit 1104 is further configured to generate n candidate feature tensors according to the n sets of weighting coefficients and a plurality of sub-feature tensors in the n sub-feature tensors, where one set of weighting coefficients corresponds to one candidate feature tensor.

[0125] In an implementation, the third determining unit 1104 is further configured to determine a sub-feature tensor with the largest weight in the plurality of sub-feature tensors for generating each candidate feature tensor, to obtain n sub-feature tensors with the largest weight.

[0126] In an implementation, the updating unit 1105 is specifically configured to determine the number k of sub-feature tensors that are different from each other in the sub-feature tensors corresponding to the n maximum values, where k is a positive integer less than or equal to n, and the updated output channel number of the convolutional layer is kN / n.

[0127] Figure 12 is a schematic structural block diagram of an image processing apparatus provided by an embodiment of the present application, wherein the image processing apparatus 1200 includes:

[0128] The obtaining unit 1201 is configured to obtain an image to be processed.

[0129] The classification unit 1202 is configured to classify the image to be processed according to a target neural network, to obtain a classification result of the image to be processed, where the channel number of the target neural network is determined by the apparatus 1100.

[0130] Those skilled in the art can understand that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solutions. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0131] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0132] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0133] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0134] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0135] If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the essential part or part of the technical solutions that make contributions to the prior art can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0136] The above merely provides the specific implementation of the present application, but the protection scope of the present application is not limited to this. Any person skilled in the art can easily think of the changes or replacements within the technical range disclosed by the present application, which should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image processing method, characterized by, The method comprises: obtaining an image to be processed; classifying the image to be processed according to a target neural network to obtain a classification result of the image to be processed; wherein the determination of the channel number of the target neural network comprises: determining an output channel number N of a convolutional layer, N being a positive integer; dividing a feature tensor output by the convolutional layer into n sub-feature tensors, each sub-feature tensor having a channel number of N / n, N / n and n being integers, and n≥2; determining n sets of weighting coefficients, each set of weighting coefficients comprising a plurality of weighting coefficients, the plurality of weighting coefficients corresponding to a plurality of sub-feature tensors in the n sub-feature tensors one by one; determining a sub-feature tensor corresponding to a maximum value in each set of weighting coefficients in the n sets of weighting coefficients to serve as n target sub-feature tensors; determining a number k of sub-feature tensors that are different from each other in the n target sub-feature tensors, k being a positive integer less than or equal to n; updating the output channel number of the convolutional layer to kN / n.

2. The method of claim 1, wherein, Each set of weighting coefficients comprises n weighting coefficients, and the n weighting coefficients correspond to the n sub-feature tensors one by one.

3. The method of claim 1, wherein, Each set of weighting coefficients comprises m weighting coefficients, and the m weighting coefficients correspond to m sub-feature tensors in the n sub-feature tensors one by one, where m is a positive integer less than n.

4. The method according to any one of claims 1 to 3, characterized in that, Before the determination of the sub-feature tensor corresponding to the maximum value in each set of weighting coefficients in the n sets of weighting coefficients, the method further comprises: generating n candidate feature tensors according to the n sets of weighting coefficients and a plurality of sub-feature tensors in the n sub-feature tensors, wherein one set of weighting coefficients corresponds to one candidate feature tensor.

5. The method of claim 4, wherein, The determination of the sub-feature tensor corresponding to the maximum value in each set of weighting coefficients in the n sets of weighting coefficients to serve as n target sub-feature tensors comprises: determining a sub-feature tensor with the maximum weight in the plurality of sub-feature tensors used to generate each candidate feature tensor to serve as the n target sub-feature tensors.

6. An image processing apparatus characterized by comprising: The method comprises: a memory for storing a program; a processor for executing the program stored in the memory, when the program stored in the memory is executed, the processor is configured to perform the following processes: obtaining an image to be processed; classifying the image to be processed according to a target neural network to obtain a classification result of the image to be processed; wherein the determination of the channel number of the target neural network comprises: determining an output channel number N of a convolutional layer, N being a positive integer; dividing a feature tensor output by the convolutional layer into n sub-feature tensors, each sub-feature tensor having a channel number of N / n, N / n and n being integers, and n≥2; determining n sets of weighting coefficients, each set of weighting coefficients comprising a plurality of weighting coefficients, the plurality of weighting coefficients corresponding to a plurality of sub-feature tensors in the n sub-feature tensors one by one; determining a sub-feature tensor corresponding to a maximum value in each set of weighting coefficients in the n sets of weighting coefficients to serve as n target sub-feature tensors; determining a number k of sub-feature tensors that are different from each other in the n target sub-feature tensors, k being a positive integer less than or equal to n; updating the output channel number of the convolutional layer to kN / n.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores program codes for execution by the device, the program codes comprising instructions for performing the method of any one of claims 1 to 5.

8. A chip, characterized by The chip comprises a processor and a data interface, the processor reading instructions stored on a memory through the data interface to perform the method of any one of claims 1 to 5.

Images