Method, device and system for neural network structure search

Abstract

The application relates to the field of artificial intelligence, in particular to the field of computer vision, and provides a neural network structure searching method, device and system. The method comprises the following steps: determining the channel numbers of all convolution layers in a convolutional neural network, wherein the channel number of a first convolution layer is proportional to the channel number of a second convolution layer; constructing a search space of the convolutional neural network according to the channel numbers of all the convolution layers; and searching the neural network structure of the convolutional neural network according to the search space of the convolutional neural network. By setting a proportional constraint on the channel numbers of the convolution layers, the total number of combinations of the channel numbers can be reduced, so that the search space of the convolutional neural network is reduced, and therefore the calculation amount of the convolutional neural network search is reduced.

Description

Technical Field

[0001] This application relates to the field of computer vision, and in particular to a method, apparatus and system for searching neural network structures. Background Technology

[0002] Neural architecture search (NAS) is a very popular subfield within automated machine learning. NAS refers to using algorithms to automatically search for the optimal neural network structure from a large pool of neural network structures, replacing tedious manual operations; in other words, it involves the machine automatically searching for the best neural network structure. Convolutional neural network architecture search, as the name suggests, is the search for neural network structures specifically for convolutional neural networks.

[0003] Currently, the search for convolutional neural network architectures suffers from the problem of enormous computational cost. Summary of the Invention

[0004] This application provides a method, apparatus, and system for searching neural network structures. By setting a proportional constraint on the number of channels in a convolutional layer, the total number of channel combinations can be reduced, thereby reducing the search space of the convolutional neural network and thus reducing the computational load of the convolutional neural network search.

[0005] In a first aspect, a method for searching neural network structures is provided, the method comprising: determining the number of channels in each convolutional layer of a convolutional neural network, wherein the number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer; constructing a search space of the convolutional neural network based on the number of channels in each convolutional layer; and searching for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network.

[0006] The number of channels in the convolutional layer mentioned in the technical solution provided in this application refers to the number of output channels of the convolutional layer.

[0007] The number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer, meaning that the number of channels in the first convolutional layer changes proportionally to the number of channels in the second convolutional layer, or vice versa.

[0008] The number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer. This can also be expressed as the first and second convolutional layers having a channel number constraint.

[0009] By setting a proportional constraint on the number of channels in a convolutional layer, the total number of channel combinations can be reduced, thereby reducing the search space of the convolutional neural network and thus reducing the computational cost of searching the convolutional neural network.

[0010] Optionally, each convolutional layer in a convolutional neural network has channel number constraints with other convolutional layers.

[0011] By setting each convolutional layer in a convolutional neural network to have channel number constraints with other convolutional layers, the total number of channel number combinations can be further reduced, thus further reducing the computational cost of convolutional neural network search.

[0012] Optionally, some convolutional layers of a convolutional neural network have channel number constraints.

[0013] The first convolutional layer can have channel number constraints with one or more other convolutional layers.

[0014] In conjunction with the first aspect, in one possible implementation of the first aspect, the convolutional layers of the convolutional neural network include a third convolutional layer, the number of channels of the third convolutional layer being proportional to the number of channels of the first convolutional layer.

[0015] By setting channel number constraints for the same convolutional layer and multiple convolutional layers, the total number of channel combinations can be further reduced, thereby further reducing the search space of the convolutional neural network and thus further reducing the computational cost of the convolutional neural network search.

[0016] The first and second convolutional layers can be continuous or discontinuous. In other words, two convolutional layers with channel number constraints can be continuous or discontinuous.

[0017] In conjunction with the first aspect, in one possible implementation of the first aspect, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in the convolutional neural network.

[0018] By setting channel number constraints on two consecutive convolutional layers in a convolutional neural network, the basic shape of the convolutional neural network structure can be controlled by setting the channel number ratio of the convolutional layers, which is beneficial for building a better convolutional neural network structure.

[0019] In conjunction with the first aspect, in one possible implementation of the first aspect, the first convolutional layer is the convolutional layer preceding the second convolutional layer; the convolutional layers of the convolutional neural network include a fourth convolutional layer, which is the convolutional layer following the second convolutional layer; wherein the ratio of the number of channels of the second convolutional layer to the number of channels of the fourth convolutional layer is equal to the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer.

[0020] By setting a uniform channel ratio for every two consecutive convolutional layers in a convolutional neural network, it is possible to better control the basic shape of the convolutional neural network structure, which is beneficial for building better convolutional neural network structures.

[0021] In conjunction with the first aspect, in one possible implementation of the first aspect, determining the number of channels in each convolutional layer of the convolutional neural network includes: randomly selecting a first value from a preset numerical range as the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer; and determining the number of channels of the first convolutional layer as a second value based on the product of the first value and the number of channels of the second convolutional layer.

[0022] It should be understood that the second value is an integer. If the product of the first value and the number of channels in the second convolutional layer is an integer, the product of the first value and the number of channels in the second convolutional layer can be used as the second value. If the product of the first value and the number of channels in the second convolutional layer is a decimal, the product of the first value and the number of channels in the second convolutional layer can be rounded down to obtain the second value. The rounding process can be either rounding up or rounding down.

[0023] In conjunction with the first aspect, in one possible implementation of the first aspect, before constructing the search space of the convolutional neural network based on the number of channels of each convolutional layer, the method further includes: randomly selecting a third value near the second value, and updating the number of channels of the first convolutional layer to the third value.

[0024] Optionally, a third value may be randomly selected from values ​​whose absolute value of the difference from the second value is within the threshold range.

[0025] Optionally, the third value can be a multiple of the second value.

[0026] The rule for deriving the third value from the second value can be flexibly determined according to specific needs.

[0027] The channel ratio of multiple convolutional layers with channel number constraints can be randomly selected, which is beneficial for constructing convolutional neural network structures of different shapes, thereby enabling a thorough search of convolutional neural network structures.

[0028] In conjunction with the first aspect, in one possible implementation of the first aspect, constructing the search space of the convolutional neural network based on the number of channels of each convolutional layer includes: constructing the search space of the convolutional neural network based on the number of channels of each convolutional layer, the number of layers of the convolutional neural network, and the size of the convolutional kernels on each convolutional layer.

[0029] For each convolutional layer with channel number constraints, the number of channels can be randomly selected from the vicinity of the value calculated based on the channel number ratio. This helps to achieve a thorough search of the convolutional neural network structure.

[0030] In conjunction with the first aspect, in one possible implementation of the first aspect, the convolutional neural network is applied to an image processing task; wherein, the step of searching for the neural network structure of the convolutional neural network according to the search space of the convolutional neural network includes: using the training data of the image processing task, searching for the neural network structure of the convolutional neural network according to the search space of the convolutional neural network.

[0031] Secondly, an apparatus for searching neural network structures is provided, the apparatus comprising a determining unit, a constructing unit, and a searching unit. The determining unit is used to determine the number of channels in each convolutional layer of a convolutional neural network, wherein the number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer. The constructing unit is used to construct a search space for the convolutional neural network based on the number of channels in each convolutional layer. The searching unit is used to search for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network.

[0032] In conjunction with the second aspect, in one possible implementation of the second aspect, the convolutional layers of the convolutional neural network include a third convolutional layer, the number of channels of the third convolutional layer being proportional to the number of channels of the first convolutional layer.

[0033] In conjunction with the second aspect, in one possible implementation of the second aspect, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in the convolutional neural network.

[0034] In conjunction with the second aspect, in one possible implementation of the second aspect, the first convolutional layer is the convolutional layer preceding the second convolutional layer; the convolutional layers of the convolutional neural network include a fourth convolutional layer, which is the convolutional layer following the second convolutional layer; wherein, the ratio of the number of channels of the second convolutional layer to the number of channels of the fourth convolutional layer is equal to the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer.

[0035] In conjunction with the second aspect, in one possible implementation of the second aspect, the determining unit is configured to: randomly select a first value from a preset numerical range as the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer; and determine the number of channels of the first convolutional layer as a second value based on the product of the first value and the number of channels of the second convolutional layer.

[0036] In conjunction with the second aspect, in one possible implementation of the second aspect, the determining unit is further configured to, before the constructing unit constructs the search space of the convolutional neural network based on the number of channels of each convolutional layer, randomly select a third value near the second value and update the number of channels of the first convolutional layer to the third value.

[0037] In conjunction with the second aspect, in one possible implementation of the second aspect, the construction unit is used to construct the search space of the convolutional neural network based on the number of channels of each convolutional layer, the number of layers of the convolutional neural network, and the size of the convolutional kernels on each convolutional layer.

[0038] In conjunction with the second aspect, in one possible implementation of the second aspect, the convolutional neural network is applied to an image processing task; wherein the search unit is used to search for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network, using the training data of the image processing task.

[0039] Thirdly, a data processing apparatus is provided, comprising: a memory for storing a program; and a processor for executing the program stored in the memory, wherein when the program stored in the memory is executed, the processor executes the method provided in the first aspect.

[0040] Fourthly, a server device is provided, including a storage module, a neural network structure search apparatus as provided in the second aspect, and an output module. The storage module is used to store training data required for convolutional neural network structure search. The neural network structure search apparatus is used to perform neural network structure search on the convolutional neural network using the training data stored in the storage module. The output module is used to output the convolutional neural network structure searched by the neural network structure search apparatus.

[0041] Fifthly, a system for searching neural network structures is provided, comprising a neural network structure running platform and a server device provided in the fourth aspect. The server device is used to deploy the searched convolutional neural network structure onto the neural network structure running platform. The neural network structure running platform is used to run the deployed convolutional neural network structure.

[0042] In a sixth aspect, a computer-readable medium is provided that stores program code for execution by a device, the program code including methods provided in the first aspect above.

[0043] In a seventh aspect, a computer program product containing instructions is provided, which, when run on a computer, causes the computer to perform the method provided in the first aspect.

[0044] Eighthly, a chip is provided, the chip including a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface and executes the method provided in the first aspect.

[0045] Optionally, as one implementation, the chip may further include a memory storing instructions, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to perform the method provided in the first aspect above.

[0046] As can be seen from the above description, this application can reduce the total number of channel combinations by setting a proportional constraint on the number of channels in the convolutional layer, thereby reducing the search space of the convolutional neural network and thus reducing the computational cost of searching the convolutional neural network. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of a 4-layer convolutional neural network structure.

[0048] Figure 2 This is a schematic block diagram of a convolutional neural network architecture search system that can be applied to the embodiments of this application.

[0049] Figure 3 for Figure 2 A schematic diagram of the convolutional neural network architecture search library 213.

[0050] Figure 4 This is a schematic flowchart illustrating a method for searching neural network structures provided in an embodiment of this application.

[0051] Figure 5 This is a schematic diagram of a convolutional neural network structure.

[0052] Figure 6 Another schematic flowchart of a method for searching neural network structures provided in an embodiment of this application.

[0053] Figure 7 This is a schematic flowchart illustrating the method for obtaining the number of channels on a convolutional layer in an embodiment of this application.

[0054] Figure 8 This is an illustrative flowchart of the method for randomly generating cross-layer connections in the embodiments of this application.

[0055] Figure 9 This is a comparison of the simulation results of the convolutional neural network structure searched using the embodiments of this application and the convolutional neural network structure searched manually in a super-resolution scene.

[0056] Figure 10 A schematic block diagram of a neural network structure search apparatus provided in an embodiment of this application.

[0057] Figure 11 Another schematic block diagram of the apparatus for searching neural network structures provided in the embodiments of this application.

[0058] Figure 12 Another schematic block diagram of the apparatus for searching neural network structures provided in the embodiments of this application.

[0059] Figure 13 for Figure 12 The diagram shows a schematic flowchart of the device performing convolutional neural network architecture search.

[0060] Figure 14 A schematic block diagram of a server device provided in an embodiment of this application.

[0061] Figure 15 A schematic block diagram of a neural network architecture search system provided in an embodiment of this application.

[0062] Figure 16 This is a schematic diagram of a chip hardware structure provided in an embodiment of this application. Detailed Implementation

[0063] Machine learning is a branch of artificial intelligence (AI). Deep learning is a subfield of machine learning, specifically an algorithm that uses artificial neural networks (ANNs) as its architecture to learn representations of data. An artificial neural network is a mathematical or computational model used in machine learning and cognitive science to mimic the structure and function of biological neural networks (e.g., the central nervous system of animals, particularly the brain), for estimating or approximating functions. Artificial neural networks can be simply referred to as neural networks (NNs) or neural network-like systems.

[0064] In the early stages of deep learning, the neural network structures required are typically designed manually. However, manual design presents several challenges: it requires highly specialized skills from the designer; it demands a significant investment of time in searching for the optimal neural network structure; and because the search space for neural network structures is usually very large, manual searching is extremely limited, making it difficult to find the optimal structure. Therefore, manually designing neural network structures is characterized by high labor and time costs, and a low probability of achieving optimal results.

[0065] Due to the aforementioned problems with manually designed neural network structures, automated machine learning (AutoML) has become increasingly popular, with neural architecture search (NAS) being a particularly hot subfield. Neural architecture search refers to using algorithms to replace tedious manual operations and automatically search for the optimal neural network structure from a large pool of available structures; in other words, the machine automatically searches for the best neural network structure. The process typically involves generating, training, and evaluating a large number of possible neural network structures, and then selecting the optimal one.

[0066] Convolutional neural network architecture search, as the name suggests, is the search for neural network architectures for convolutional neural networks (CNNs), that is, the automated search for convolutional neural network architectures.

[0067] A convolutional neural network (CNN) is a neural network that contains convolutional layers. The main characteristics of a CNN include the number of layers, the kernel size of each convolutional layer, the number of channels in each convolutional layer, and cross-layer connections. Therefore, when searching for a CNN architecture, the primary focus is on the number of layers, kernel size, number of channels, and cross-layer connections. For example, in some cases, the number of layers can be predetermined. In this case, the search for the CNN architecture mainly focuses on other key features, such as kernel size, number of channels, and cross-layer connections.

[0068] As an example, Figure 1 This is a schematic diagram of a 4-layer convolutional neural network structure. It should be understood that the number of layers in a convolutional neural network does not include the input and output layers. A network with 4 layers means that there are 4 convolutional layers (e.g., ...) between the input and output layers. Figure 1 Convolutional layer 1, convolutional layer 2, convolutional layer 3, and convolutional layer 4 (in the text). Figure 1 Each convolutional layer in the model can be a two-dimensional convolutional layer (Conv2D). The parameters of each convolutional layer are as follows: Figure 1 As shown, taking convolutional layer 1 as an example, the kernel size <3×3×1×11> indicates that the kernel size is 3×3, the number of input channels is 1, and the number of output channels is 11; the bias... <11> This indicates a bias of 11. In Figure 1In the example, convolutional layers 1, 2, and 3 are biased convolutional layers, while convolutional layer 4 is a bias-free convolutional layer. Figure 1 In this context, taking cross-layer connections as an example, a concatenation means that the input layer is connected to convolutional layer 1 to form the next convolutional layer (i.e., ...). Figure 1 The input to the convolutional layer 2 shown in the figure.

[0069] exist Figure 1 In the example, assuming the number of layers in the network is preset, the convolutional neural network search primarily searches for the kernel size of the convolutional layers, the number of channels in the convolutional layers, and cross-layer connections. Here, the number of channels in a convolutional layer refers to the number of output channels.

[0070] Figure 1 This is merely an example of a convolutional neural network architecture and not a limitation.

[0071] For example, the number of layers in a convolutional neural network structure can be fewer or more.

[0072] It should be understood that cross-layer connections include two methods: concatenation and addition. Concatenation means joining the outputs of multiple layers together as the input of another layer. Addition means adding the outputs of multiple layers together as the input of another layer. Taking a cross-layer connection between two layers as an example: If the cross-layer connection between two layers is concatenation, assuming the first layer has 'a' output channels and the second layer has 'b' output channels, then the outputs of these two layers are concatenated, forming a total of 'a+b' channels, which are then used as the input of another layer. If the cross-layer connection between two layers is addition, then the output channels of these two layers are the same, assuming both layers have 'a' output channels. Then the outputs of these two layers are added together, resulting in an output channel count of 'a', which is also used as the input of another layer.

[0073] exist Figure 1 In this example, a cross-layer connection is taken as the connection between the input layer and convolutional layer 1. This embodiment of the application does not limit this. For example, a cross-layer connection can also be applied to multiple other layers. The cross-layer connection method can be concatenation or addition (the output channel number of the multiple layers being added must be consistent).

[0074] In the existing search for convolutional neural network (CNN) architectures, the search space is enormous. Taking a 5-layer CNN as an example, assuming there are two possible kernel sizes (e.g., 1×1 and 3×3) and 20 possible channel numbers (e.g., channel numbers ranging from [5, 24]), then there are at least 10 possible cross-layer connections (4+3+2+1=10), a total of 32 possible kernel sizes (2 to the power of 5), and a total of 3.2 million possible channel numbers (20 to the power of 5). The total number of possible CNNs is 1.024 billion (10×32×3.2 million=1.024 billion), meaning the maximum possible search space for CNN architectures is 1.024 billion.

[0075] It should be understood that the search space of neural network architectures is very large, which leads to a huge amount of computation for convolutional neural network searches, and also requires a large number of processor modules (e.g., graphics processing units (GPUs)) to perform convolutional neural network searches.

[0076] To address the aforementioned technical problems, embodiments of this application provide a method and apparatus for searching neural network structures. By setting channel number constraints on convolutional layers in a convolutional neural network, the search space of the convolutional neural network can be significantly reduced, thereby reducing the computational load of searching the convolutional neural network structure and correspondingly reducing the number of GPUs required to perform the search.

[0077] To facilitate understanding of the embodiments of this application, an application scenario of the embodiments of this application will be described below by way of example.

[0078] Figure 2 This is a schematic block diagram of a convolutional neural network architecture search system 200 that can be applied to embodiments of this application. The convolutional neural network architecture search system 200 includes a model training server 210 and a model running platform 220.

[0079] The model training server 210 is used to search for the neural network structure of the convolutional neural network and deploy the searched convolutional neural network structure to the model running platform 220.

[0080] The model training server 210 is the carrier of data and software, and also the hardware foundation for software operation.

[0081] The model running platform 220 is used to run the deployed convolutional neural network architecture.

[0082] The model runtime platform 220 can be a terminal or other application platform. When the model runtime platform 220 is a terminal, the system 200 can be called a terminal-server system. For example, the model runtime platform 220 is a mobile phone.

[0083] like Figure 2 As shown, the model training server 210 includes a training set 211, a test set 212, a convolutional neural network structure search library 213, a script file 214, and a convolutional neural network structure file 215.

[0084] The Convolutional Neural Network Structure Search Library 213 is used for searching convolutional neural network structures.

[0085] Script file 214 is used to store relevant parameters. In the application, script file 214 can be used to call the convolutional neural network structure search library 213.

[0086] The training set 211 and the test set 212 are used to provide the datasets required for the convolutional neural network architecture search library 213 to perform convolutional neural network architecture search.

[0087] In machine learning, samples are generally divided into three parts: training set, validation set, and test set. The training set is used to build the model, and the test set is used to evaluate the performance of the ultimately selected optimal model.

[0088] Convolutional Neural Network Structure File 215 represents the neural network structure output by the convolutional neural network structure search library 213, i.e., the optimal convolutional neural network structure found. Convolutional Neural Network Structure File 215 represents the searched convolutional neural network structure in file format.

[0089] It can be understood that the model training server 210 deploys the convolutional neural network structure file 215 to the model running platform 220.

[0090] As an example, such as Figure 3 As shown, the convolutional neural network structure search library 213 includes a main controller 301, a convolutional neural network structure generator 302, a model converter 303, and a model trainer 304.

[0091] The main controller 301 of the convolutional neural network architecture search library is used to control the convolutional neural network architecture generator 302, the model converter 303, and the model trainer 304. The main controller 301 of the convolutional neural network architecture search library is called by script file 214.

[0092] The convolutional neural network structure generator 302 is used to generate convolutional neural network structures. This can be understood as the convolutional neural network structure generator 302 being used to construct the search space for convolutional neural network structures.

[0093] The model converter 303 is used to convert the convolutional neural network structure generated by the convolutional neural network structure generator 302 into a convolutional neural network model.

[0094] The model trainer 304 is used to train the convolutional neural network model obtained by the model converter 303 in order to obtain the evaluation index of the convolutional neural network model.

[0095] As an example, the process by which the model training server 210 searches for the convolutional neural network structure file 215 is as follows.

[0096] 1) Obtain the dataset, including training set 211 and test set 212.

[0097] For example, you can obtain the dataset required for a specific scenario.

[0098] 2) Set the relevant parameters in script file 214. These parameters represent the parameters required for the convolutional neural network structure search library 213 to run.

[0099] For example, the relevant parameters set in script file 214 include the following parameters:

[0100] The path to training set 211, the path to test set 212, the number of layers in the convolutional neural network structure, the total number of parameters, and the total time used for searching the convolutional neural network structure.

[0101] It should be understood that relevant parameters can be set in script file 214 as needed based on the operation of convolutional neural network architecture search library 213. For example, in addition to the parameters mentioned above, the relevant parameters set in script file 214 may also include other parameters. Furthermore, the relevant parameters set in script file 214 may not include one or more of the parameters mentioned above. This application does not impose any limitations on this.

[0102] 3) The convolutional neural network structure search library 213 is called through script file 214 to perform automated convolutional neural network structure search.

[0103] For example, when script file 214 calls convolutional neural network structure search library 213, it passes the relevant parameters mentioned in step 2) to convolutional neural network structure search library 213. Convolutional neural network structure search library 213 performs convolutional neural network structure search according to the relevant parameters set in script file 214.

[0104] 4) After the search of the convolutional neural network structure search library 213 is completed, the convolutional neural network structure search library 213 outputs the optimal convolutional neural network structure, namely the convolutional neural network structure file 215, and the model training server 210 deploys the convolutional neural network structure file 215 on the model running platform 220.

[0105] For example, when the total time for searching the convolutional neural network structure is exhausted, the convolutional neural network structure search library 213 ends the convolutional neural network structure search and outputs the convolutional neural network structure file 215 obtained during the search process.

[0106] The embodiments of this application will now be described with reference to the accompanying drawings.

[0107] Figure 4 This is a schematic flowchart illustrating a method 400 for searching neural network structures provided in an embodiment of this application. For example, the execution entity of method 400 is... Figure 2 The convolutional neural network architecture search library 213 is shown. Method 400 includes steps S410, S420, and S430.

[0108] S410, determine the number of channels in each convolutional layer of the convolutional neural network, wherein the number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer.

[0109] This convolutional neural network represents the neural network for which a convolutional neural network structure search is to be performed.

[0110] The number of channels in the convolutional layer mentioned in the embodiments of this application refers to the number of output channels of the convolutional layer.

[0111] The number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer, meaning that the number of channels in the first convolutional layer changes proportionally to the number of channels in the second convolutional layer, or vice versa.

[0112] For example, let R1 be the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer. If the number of channels in the second convolutional layer is c1, then the number of channels in the first convolutional layer is R1×c1. If the number of channels in the second convolutional layer is c2, then the number of channels in the first convolutional layer is R1×c2.

[0113] For the sake of brevity, the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer will be referred to as the channel ratio of the first convolutional layer to the second convolutional layer.

[0114] The channel ratio between the first and second convolutional layers can be defined as the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer, or the ratio of the number of channels in the second convolutional layer to the number of channels in the first convolutional layer. In practical applications, the channel ratio can be determined based on the relative positions of the first and second convolutional layers.

[0115] The number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer. This can also be expressed as the first and second convolutional layers having a channel number constraint.

[0116] For example, step S410 includes: obtaining the number of channels of the first convolutional layer based on the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer and the number of channels of the second convolutional layer.

[0117] As an example, suppose the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer is defined as the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer, and this ratio is 1.2. If the number of channels in the second convolutional layer is 5, the number of channels in the first convolutional layer is calculated to be 6; if the number of channels in the second convolutional layer is 3, the product of 3 and 1.2 (3.6) is rounded to the nearest integer 4, which is taken as the number of channels in the first convolutional layer.

[0118] It should be understood that the product of the number of channels in a convolutional layer x and the corresponding ratio of channel numbers may be a decimal. In this case, the integer nearest to the decimal is taken as the number of channels in another convolutional layer y (i.e., the convolutional layer that has a channel number constraint with convolutional layer x). There are various ways to obtain the number of channels in convolutional layer y from the decimal, such as rounding down or rounding up, etc., and the embodiments of this application do not limit this.

[0119] The first convolutional layer can represent each convolutional layer in a subset of the convolutional layers of a convolutional neural network, or it can represent each convolutional layer in the convolutional neural network. In other words, in the embodiments of this application, at least some convolutional layers in the convolutional neural network are configured to have channel number constraints.

[0120] Optionally, in Figure 4 In the illustrated embodiment, the first convolutional layer represents each convolutional layer in the partial convolutional layers of the convolutional neural network, and the second convolutional layer represents another convolutional layer in the partial convolutional layers besides the first convolutional layer.

[0121] Alternatively, the solution in this embodiment can also be described as follows: some convolutional layers in the convolutional neural network have channel number constraints. Having channel number constraints in some convolutional layers of the convolutional neural network means that for each convolutional layer in the convolutional neural network, it has a channel number constraint with at least one other convolutional layer.

[0122] As an example, a convolutional neural network consists of five convolutional layers: convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layers 1 and 2 have channel number constraints. Assume there are 20 possible channel numbers for this convolutional neural network. In this example, the total number of channel combinations for this convolutional neural network is 20 × 1 × 20 × 20 × 20 (160,000 possibilities).

[0123] Referring to the example of the prior art described above, in the case of a convolutional neural network including 5 convolutional layers and 20 possible channel sizes, the total number of channel combinations is equal to 20×20×20×20×20 (3.2 million).

[0124] It can be understood that, compared with the prior art, under the same conditions, the embodiments of this application can greatly reduce the total number of channel combinations, thereby reducing the search space of the convolutional neural network and thus reducing the computational load of the convolutional neural network search.

[0125] Optionally, in Figure 4 In the illustrated embodiment, the first convolutional layer represents each convolutional layer of the convolutional neural network, and the second convolutional layer represents another convolutional layer in the convolutional neural network besides the first convolutional layer.

[0126] Alternatively, the solution in this embodiment can also be described as follows: all convolutional layers in the convolutional neural network have channel number constraints. Having channel number constraints in all convolutional layers of the convolutional neural network means that for each convolutional layer in the convolutional neural network, it has a channel number constraint with at least one other convolutional layer.

[0127] As an example, a convolutional neural network (CNN) consists of five convolutional layers: convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layer 1 and convolutional layer 2 have channel number constraints, as do convolutional layer 1 and convolutional layer 3, and convolutional layer 4 and convolutional layer 5. Assume that the number of channels in this CNN can be selected in 20 different ways. In this example, the total number of channel combinations for this CNN is 20 × 1 × 1 × 20 × 1 (400 possibilities).

[0128] Referring to the example of the prior art described above, in the case of a convolutional neural network including 5 convolutional layers and 20 possible channel sizes, the total number of channel combinations is equal to 20×20×20×20×20 (3.2 million).

[0129] It can be understood that, compared with the prior art, under the same conditions, the embodiments of this application can effectively reduce the total number of channel combinations, thereby reducing the search space of the convolutional neural network and thus reducing the computational load of the convolutional neural network search.

[0130] As described above, in the embodiments of this application, at least some convolutional layers in the convolutional neural network have channel number constraints, that is, each convolutional layer in at least some of the convolutional layers in the convolutional neural network has a channel number constraint with another convolutional layer. The pairing method of the convolutional layers with channel number constraints in at least some of the convolutional layers can be flexibly set according to requirements, and this embodiment of the application does not limit this.

[0131] S420 constructs the search space of the convolutional neural network based on the number of channels in each convolutional layer.

[0132] For example, the search space of a convolutional neural network can be constructed based on the number of channels in each convolutional layer, the number of layers in the convolutional neural network, and the kernel size in each convolutional layer. The number of layers in the convolutional neural network can be preset.

[0133] It should be noted that the implementation of constructing the search space of the convolutional neural network based on these parameters after determining the number of layers, the number of channels in each convolutional layer, and the kernel size is existing technology and will not be described in detail in this application.

[0134] S430, based on the search space of the convolutional neural network, search for the neural network structure of the convolutional neural network.

[0135] For example, various search strategies can be employed to search for the neural network structure of a convolutional neural network based on its search space.

[0136] It should also be noted that the implementation method of searching for the neural network structure of the convolutional neural network based on this search space is existing technology, and this application will not describe it in detail.

[0137] This application embodiment reduces the total number of channel combinations by setting a proportional constraint on the number of channels in the convolutional layer, thereby reducing the search space of the convolutional neural network and thus reducing the computational load of the convolutional neural network search.

[0138] Furthermore, by setting each convolutional layer in the convolutional neural network to have channel number constraints with other convolutional layers, the total number of channel number combinations can be further reduced, thus further reducing the computational cost of convolutional neural network search.

[0139] It should be understood that, compared to existing technologies, the embodiments of this application can reduce the computational load of convolutional neural network (CNN) search, and therefore, the number of GPUs required to perform CNN architecture search is also smaller. For example, in some application scenarios, a better CNN architecture can be found using only one GPU in 8 hours.

[0140] In the embodiments of this application, a convolutional layer may have channel number constraints with one or more other convolutional layers.

[0141] Optionally, in Figure 4 In the illustrated embodiment, the convolutional layers of the convolutional neural network include a third convolutional layer, and the number of channels in the third convolutional layer is also proportional to the number of channels in the first convolutional layer. In other words, the number of channels in the first convolutional layer is proportional to the number of channels in both the second and third convolutional layers.

[0142] In this embodiment, the first convolutional layer has channel number constraints with both the second and third convolutional layers. It can be understood that the second and third convolutional layers also have channel number constraints.

[0143] It should be noted that the statement in this embodiment that "the first convolutional layer has channel number constraints with the second and third convolutional layers respectively" indicates that the first convolutional layer can have channel number constraints with multiple convolutional layers respectively, and does not mean that the first convolutional layer can only have channel number constraints with two convolutional layers.

[0144] It should be understood that the more convolutional layers that have channel number constraints with the first convolutional layer, the more options there are for the number of channels in the convolutional layers.

[0145] As an example, a convolutional neural network (CNN) consists of five convolutional layers: Convolutional Layer 1, Convolutional Layer 2, Convolutional Layer 3, Convolutional Layer 4, and Convolutional Layer 5. Convolutional Layer 1 has channel number constraints with Convolutional Layers 2 and 3, respectively. Assume there are 20 possible channel numbers for this CNN. In this example, the total number of channel combinations for this CNN is 20 × 1 × 1 × 20 × 20 (8000). Convolutional Layers 2 and 3 have only one channel number option.

[0146] As another example, a convolutional neural network (CNN) consists of five convolutional layers: convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layer 1 has channel number constraints with convolutional layers 2, 3, and 4, respectively. Assume there are 20 possible channel numbers for this CNN. In this example, the total number of channel combinations for this CNN is 20 × 1 × 1 × 1 × 20 (400 possibilities). Convolutional layers 2, 3, and 4 have only one channel number option.

[0147] In this embodiment of the application, by setting the same convolutional layer and multiple convolutional layers to have channel number constraints, the total number of channel number combinations can be further reduced, thereby further reducing the search space of the convolutional neural network and thus further reducing the computational cost of convolutional neural network search.

[0148] Optionally, in Figure 4 In the illustrated embodiment, within a convolutional layer with channel number constraints, each convolutional layer may also have channel number constraints with only one other convolutional layer.

[0149] As an example, a convolutional neural network (CNN) consists of five convolutional layers: Convolutional Layer 1, Convolutional Layer 2, Convolutional Layer 3, Convolutional Layer 4, and Convolutional Layer 5. Convolutional Layers 1 and 2 have channel number constraints, as do Convolutional Layers 4 and 5. Assume there are 20 possible channel numbers for this CNN. In this example, the total number of channel combinations for this CNN is 20 × 1 × 20 × 20 × 1 (8000 possibilities).

[0150] It should be understood that, compared to the prior art, this embodiment can also reduce the total number of channel combinations, thereby reducing the search space of the convolutional neural network and thus reducing the computational cost of the convolutional neural network search.

[0151] In the embodiments of this application, the first convolutional layer and the second convolutional layer may be continuous or discontinuous. In other words, two convolutional layers with channel number constraints may be continuous or discontinuous.

[0152] Two consecutive convolutional layers mean that one of the two convolutional layers is the convolutional layer preceding the other (or, one is the convolutional layer following the other), meaning that there are no other convolutional layers between the two convolutional layers.

[0153] Two non-contiguous convolutional layers indicate that there are one or more other convolutional layers between these two convolutional layers.

[0154] Optionally, in some embodiments, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in a convolutional neural network.

[0155] In the case where the first convolutional layer represents each convolutional layer in a convolutional neural network, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in the convolutional neural network, meaning that every two consecutive convolutional layers in the convolutional neural network have a channel number constraint.

[0156] As an example, a convolutional neural network consists of five convolutional layers: convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layer 1 precedes convolutional layer 2, convolutional layer 2 precedes convolutional layer 3, and so on, with convolutional layer 4 preceding convolutional layer 5. Convolutional layers 1 and 2 have channel number constraints, as do convolutional layers 2 and 3, 3 and 4, and 5.

[0157] On the one hand, in this example, this is equivalent to convolutional layer 1 having channel number constraints with convolutional layers 2, 3, 4, and 5, respectively. Assuming there are 20 possible channel numbers for this convolutional neural network, the total number of channel combinations is 20 × 1 × 1 × 1 × 1 (20 combinations). It can be understood that by setting channel number constraints for every two consecutive convolutional layers in the convolutional neural network, the total number of channel combinations can be significantly reduced.

[0158] On the other hand, in this example, the shape of the convolutional neural network structure is basically determined by the ratio of the number of channels between every two consecutive convolutional layers. That is, in this example, the basic shape of the convolutional neural network structure can be controlled by setting the ratio of the number of channels between every two consecutive convolutional layers, which is beneficial for building a better convolutional neural network structure.

[0159] It should be understood that, in this embodiment, by setting a channel number constraint for every two consecutive convolutional layers in the convolutional neural network, the shape of the convolutional neural network structure can be controlled by setting the channel number ratio of every two consecutive convolutional layers with channel number constraints, which is beneficial for building a better convolutional neural network structure.

[0160] In the case where the first convolutional layer represents each convolutional layer in the partial convolutional layers of a convolutional neural network, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in the convolutional neural network, meaning that every two consecutive convolutional layers in the partial convolutional layers of the convolutional neural network have channel number constraints.

[0161] It should be understood that, in this embodiment, the shape of the convolutional neural network structure can also be controlled by setting the channel ratio of every two consecutive convolutional layers with channel number constraints, which is beneficial for building a better convolutional neural network structure.

[0162] Optionally, in some embodiments, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in a convolutional neural network, and the first convolutional layer is the convolutional layer preceding the second convolutional layer; the convolutional layers of the convolutional neural network also include a fourth convolutional layer, which is the convolutional layer following the second convolutional layer; wherein, the ratio of the number of channels in the second convolutional layer to the number of channels in the fourth convolutional layer is equal to the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer.

[0163] In other words, in this embodiment, the channel count ratio of all two consecutive convolutional layers with channel count constraints in the convolutional neural network is equal (i.e., equal to the same value). Or, in this embodiment, the channel count ratio of any two consecutive convolutional layers in the convolutional neural network structure is maintained at the same level.

[0164] As an example, a convolutional neural network (CNN) consists of five convolutional layers: convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layer 1 precedes convolutional layer 2, convolutional layer 2 precedes convolutional layer 3, and so on, with convolutional layer 4 preceding convolutional layer 5. For instance, in a given 5-layer CNN architecture, if the randomly selected channel ratio (FR) is 1.2, then the number of channels in convolutional layer 1 is 1.2 times the number of channels in convolutional layer 2, the number of channels in convolutional layer 2 is 1.2 times the number of channels in convolutional layer 3, and so on, with the number of channels in convolutional layer 4 being 1.2 times the number of channels in convolutional layer 5. In this example, when calculating the number of channel combinations for each convolutional layer, only two variables are used: the channel ratio (FR) and the number of channels in convolutional layer 5.

[0165] For example, in step S410, assuming the total number of parameters of the convolutional neural network structure has been set, after the cross-layer connections and the kernel size of each convolutional layer are determined, the desired number of channels of the last convolutional layer (such as convolutional layer 5 in this example) can be calculated based on the channel ratio (FR) of the convolutional layer (e.g., denoted as filters(N)). Then, based on filters(N) and the channel ratio (FR), the number of channels of each convolutional layer can be calculated, where N represents the number of layers of the convolutional neural network (excluding the input and output layers).

[0166] For example, the total number of parameters in a convolutional neural network structure is 3993.

[0167] Figure 5 The diagram shows the channel ratio of all consecutive convolutional layers with channel number constraints in a convolutional neural network (corresponding to...). Figure 5 The diagram shown illustrates FR values ​​of 1.2, 1.0, and 0.8. In Figure 5 In this diagram, a rectangle represents a convolutional layer, and the width of the rectangle indicates the number of channels in that layer. A wider rectangle indicates a larger number of channels, and a smaller rectangle indicates a smaller number of channels. The topmost rectangle represents the first convolutional layer, and the bottommost rectangle represents the fifth convolutional layer. FR represents the ratio of the number of channels in the preceding convolutional layer to the number of channels in the following convolutional layer.

[0168] Figure 5 The diagram also shows a schematic of the shape of a convolutional neural network structure with no channel number constraint on the convolutional layers. It can be seen that without a channel number constraint, the number of channels in each convolutional layer is random, resulting in a very large total number of channel combinations across all convolutional layers, thus leading to a very large search space.

[0169] When convolutional layers have channel number constraints, the total number of channel combinations for each convolutional layer can be reduced, thereby reducing the search space.

[0170] In addition, from Figure 5 It can also be seen that when there is no channel number constraint in the convolutional layer, the number of channels in each convolutional layer is random, so the shape of the convolutional neural network structure is completely random and cannot be controlled.

[0171] In this embodiment, by setting the channel number ratio of all consecutive convolutional layers with channel number constraints in the convolutional neural network to be uniform, the shape of the convolutional neural network structure is basically determined by the channel number ratio (FR). Therefore, the shape of the convolutional neural network structure can be controlled, which is beneficial for building a better convolutional neural network structure.

[0172] Therefore, in this embodiment, by setting a uniform channel ratio for every two consecutive convolutional layers in the convolutional neural network, the shape of the convolutional neural network structure can be better controlled, which is beneficial for building a better convolutional neural network structure.

[0173] Optionally, in some embodiments, the first convolutional layer and the second convolutional layer may also be two non-contiguous convolutional layers in a convolutional neural network.

[0174] As an example, a convolutional neural network (CNN) consists of five convolutional layers: convolutional layer 1, convolutional layer 2, convolutional layer 3, convolutional layer 4, and convolutional layer 5. Convolutional layer 1 precedes convolutional layer 2, convolutional layer 2 precedes convolutional layer 3, and so on, with convolutional layer 4 preceding convolutional layer 5. Convolutional layers 1 and 3 have channel number constraints, as do convolutional layers 2 and 4, and convolutional layers 1 and 5. Assuming there are 20 possible channel numbers for this CNN, the total number of channel combinations is 20 × 20 × 1 × 1 × 1 (400 combinations).

[0175] In the embodiments of this application, the channel ratio of multiple convolutional layers with channel number constraints can be randomly selected, which can enable the construction of convolutional neural network structures of different shapes, thereby facilitating the full search for possible convolutional neural network structures.

[0176] Optionally, such as Figure 6 As shown, in Figure 4 In the illustrated embodiment, step S410 includes steps S411 and S412.

[0177] S411, randomly select a first value from a preset range as the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer.

[0178] For example, the preset value range is [0.8, 2.5]. Suppose the first value is 1.2, then the number of channels in the first convolutional layer is 1.2 times that of the second convolutional layer.

[0179] For example, in step S410, when generating a convolutional neural network structure, a value FR is randomly selected within a preset numerical range (e.g., [0.8, 2.5]). This value FR represents the ratio of the number of channels in the previous convolutional layer to the number of channels in the next convolutional layer. For example, this ensures that the ratio of the number of channels in the previous convolutional layer to the number of channels in the next convolutional layer in any two consecutive convolutional layers of the convolutional neural network is equal to the value FR.

[0180] S412, determine the number of channels of the first convolutional layer as the second value based on the product of the first value and the number of channels of the second convolutional layer.

[0181] It should be understood that the second value is an integer. If the product of the first value and the number of channels in the second convolutional layer is an integer, the second value can be equal to the product of the first value and the number of channels in the second convolutional layer. If the product of the first value and the number of channels in the second convolutional layer is not an integer, the second value can be the value obtained by rounding down the product of the first value and the number of channels in the second convolutional layer. This rounding can be either rounding up or rounding down.

[0182] In this embodiment, by randomly selecting the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer from a preset numerical range, convolutional neural network structures of different shapes can be constructed.

[0183] It should be understood that, in the embodiments of this application, the channel ratio between convolutional layers can also be preset. For example, the channel ratio between each convolutional layer can be determined according to application requirements.

[0184] In this embodiment of the application, the number of channels in a convolutional layer with channel number constraints can be randomly selected from the range of values ​​calculated based on the channel number ratio.

[0185] Optionally, such as Figure 6 As shown, in Figure 4 In the illustrated embodiment, step S410 includes the steps S411 and S412 described above. Before step S420 (constructing the search space of the convolutional neural network based on the number of channels in each convolutional layer), the method 400 further includes step S440.

[0186] S440, randomly select a third value near the second value, and update the number of channels of the first convolutional layer to the third value.

[0187] For example, a value can be randomly selected from integers whose absolute value of the difference from the second value does not exceed a threshold as the number of channels in the first convolutional layer. This threshold could be, for example, 0.5, 1, 1.5, or other values. In application, the value of this threshold can be determined according to specific requirements.

[0188] For example, a third value, which is s times the second value, can be used as the number of channels in the first convolutional layer, where the product of s and the second value is an integer. The value of s can be determined according to specific requirements.

[0189] For example, other preset rules can be used to randomly select a value as the number of channels in the first convolutional layer based on the second value.

[0190] In this embodiment, for the number of channels in a convolutional layer with channel number constraints, the number of channels can be randomly selected from the vicinity of the value calculated based on the channel number ratio. This allows for a more thorough search of possible convolutional neural network structures.

[0191] As described above, in step S420, the search space of the convolutional neural network can be constructed based on the number of channels in each convolutional layer, the number of layers in the convolutional neural network, and the size of the convolutional kernels in each convolutional layer.

[0192] The number of layers in a convolutional neural network can be preset or randomly selected.

[0193] In this context, the kernel size of each convolutional layer in a convolutional neural network can also be randomly selected.

[0194] For example, the kernel size of each convolutional layer can be randomly selected from a preset range of values.

[0195] It should be understood that randomly selecting the kernel size on each convolutional layer during the process of constructing the search space of a convolutional neural network can facilitate a thorough search of the convolutional neural network structure.

[0196] For example, step S420 further includes setting the total number of parameters of the convolutional neural network structure. For example, the total number of parameters of the convolutional neural network structure is 3993.

[0197] For example, step S420 further includes randomly generating cross-layer connections based on the number of layers in the convolutional neural network.

[0198] As an example, the illustrative process for determining the number of channels in each convolutional layer during the construction of a convolutional neural network structure is as follows: Figure 7 As shown. In Figure 7In the example, every two consecutive convolutional layers in the convolutional neural network have a channel number constraint, and all two consecutive convolutional layers with channel number constraints have the same channel number ratio (i.e., equal to the same value FR).

[0199] S710 randomly selects the kernel size for each convolutional layer of the convolutional neural network.

[0200] S720 randomly generates cross-layer connections based on the number of layers in the convolutional neural network.

[0201] S730 randomly selects the proportion of convolutional layer channels (FR) within a preset range.

[0202] The channel ratio (FR) of a convolutional layer represents the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer in any two consecutive convolutional layers in a convolutional neural network.

[0203] S740 calculates the number of channels in each convolutional layer based on the total number of parameters of the convolutional neural network structure, the number of layers, the kernel size of each convolutional layer, cross-layer connections, and the channel ratio (FR) of the convolutional layer.

[0204] S750 updates the number of channels for each convolutional layer to a value randomly selected within a certain range near the value calculated based on the channel ratio (FR) of the convolutional layer (it should be understood that this value is an integer).

[0205] For example, in the method 400 provided in the above embodiments, step S410 may include Figure 7 Steps S710 to S740, step S440 may include Figure 7 Step S750 in the process.

[0206] As an example, the illustrative flow of generating cross-layer connections during each construction of a convolutional neural network structure is as follows: Figure 8 As shown. For example, in Figure 7 In the illustrated embodiment, step S720 includes Figure 8 Steps S801 to S811 are shown.

[0207] S801, select cross-layer connection method.

[0208] For example, randomly select one of the following two methods as the cross-layer connection method:

[0209] 1) Contains only connections (concatenate);

[0210] 2) It includes both addition and concatenation.

[0211] S802, determine whether the cross-layer connection method only includes concatenation. If yes, proceed to step S803; otherwise, proceed to step S806.

[0212] S803 calculates the maximum possible number of concatenations C based on the number of layers in the convolutional neural network.

[0213] S804, randomly selects the number of cross-layer connections using the concatenate method from the range [0,C].

[0214] S805, randomly selects each connection (concatenate).

[0215] S806, calculate the maximum possible number of add(s) A based on the number of layers in the convolutional neural network.

[0216] S807, randomly select the number of sums (add) from the range [0, A].

[0217] S808 randomly selects each cross-layer connection using the add method.

[0218] S809, calculate the number C of cross-layer connections that a convolutional neural network can still perform after having add-type cross-layer connections.

[0219] S810, randomly selects the number of cross-layer connections using the concatenate method from the range [0,C].

[0220] S811 randomly selects each concatenate cross-layer connection.

[0221] As can be seen from the above description, in this embodiment of the application, by setting a proportional constraint on the number of channels in the convolutional layer, the total number of combinations of channels can be reduced, thereby reducing the search space of the convolutional neural network and thus reducing the computational load of the convolutional neural network search.

[0222] For example, embodiments of this application can be applied to neural network architecture search for convolutional neural networks used to process image data.

[0223] Optionally, Figure 4 The convolutional neural network involved in the illustrated embodiment is applied to an image processing task; wherein, step S430 includes: using the training data of the image processing task, searching for the neural network structure of the convolutional neural network according to the search space of the convolutional neural network.

[0224] For example, an image processing task is super-resolution (SR). Super-resolution is a technique that improves the resolution of an original image using hardware or software methods. For instance, the process of obtaining a high-resolution image from a series of low-resolution images or a single low-resolution image is called super-resolution reconstruction.

[0225] For example, embodiments of this application can be applied to a convolutional neural network architecture search system with three times super-resolution.

[0226] Simulation experiments using super-resolution as an example demonstrate that the scheme provided in this application for searching convolutional neural network (CNN) structures, using a single GPU, can automatically search for 364 CNN structures within 8 hours. Furthermore, the super-resolution performance using the optimal CNN structure found outperforms that using a manually designed CNN structure. The comparison is shown in the figure below. Figure 9 As shown.

[0227] exist Figure 9 In the image, the left side shows the super-resolution effect achieved using the optimal convolutional neural network structure found in this application's embodiments, while the right side shows the effect of a manually designed convolutional neural network structure. It can be seen that the left image exhibits less ringing noise at the text edges and achieves better results than the right image.

[0228] The various embodiments described herein can be independent solutions or combinations thereof based on their inherent logic, and all such solutions fall within the protection scope of this application.

[0229] The method embodiments provided in this application have been described above. The apparatus embodiments provided in this application will be described below. It should be understood that the description of the apparatus embodiments corresponds to the description of the method embodiments. Therefore, any content not described in detail can be referred to the method embodiments above. For the sake of brevity, it will not be repeated here.

[0230] like Figure 10 As shown, this application embodiment provides a neural network structure search apparatus 1000. The apparatus 1000 can be used to execute the method 400 in the above embodiment. The apparatus 1000 includes a determining unit 1010, a constructing unit 1020, and a searching unit 1030.

[0231] The determining unit 1010 is used to determine the number of channels in each convolutional layer of the convolutional neural network, wherein the number of channels in the first convolutional layer is proportional to the number of channels in the second convolutional layer.

[0232] The building unit 1020 is used to construct the search space of the convolutional neural network based on the number of channels in each convolutional layer.

[0233] The search unit 1030 is used to search for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network.

[0234] Optionally, the convolutional layers of the convolutional neural network include a third convolutional layer, and the number of channels in the third convolutional layer is also proportional to the number of channels in the first convolutional layer.

[0235] Optionally, the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in a convolutional neural network.

[0236] Optionally, the first convolutional layer is the convolutional layer preceding the second convolutional layer; the convolutional layers of the convolutional neural network include a fourth convolutional layer, which is the convolutional layer following the second convolutional layer; wherein, the ratio of the number of channels in the second convolutional layer to the number of channels in the fourth convolutional layer is equal to the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer.

[0237] Optionally, the determining unit 1010 is used to: randomly select a first value from a preset numerical range as the ratio of the number of channels of the first convolutional layer to the number of channels of the second convolutional layer; and determine the number of channels of the first convolutional layer as the second value based on the product of the first value and the number of channels of the second convolutional layer.

[0238] Optionally, the determining unit 1010 is further configured to randomly select a third value near the second value and update the number of channels of the first convolutional layer to the third value before the building unit 1020 constructs the search space of the convolutional neural network based on the number of channels of each convolutional layer.

[0239] Optionally, the construction unit 1020 is used to construct the search space of the convolutional neural network based on the number of channels in each convolutional layer, the number of layers in the convolutional neural network, and the size of the convolutional kernels in each convolutional layer.

[0240] Optionally, the convolutional neural network is applied to an image processing task; wherein, the search unit 1030 is used to search for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network, using the training data of the image processing task.

[0241] For example, the neural network structure search apparatus 1000 provided in this application embodiment is implemented as program code that can run on a computer.

[0242] like Figure 11 As shown in the figure, this application embodiment also provides a neural network structure search device 1100. The device 1100 includes a processor 1110, which is coupled to a memory 1120. The memory 1120 is used to store computer programs or instructions, and the processor 1110 is used to execute the computer programs or instructions stored in the memory 1120, so that the methods in the above method embodiments are executed.

[0243] Optionally, such as Figure 11 As shown, the device 1100 may also include a memory 1120.

[0244] Optionally, such as Figure 11 As shown, the device 1100 may also include a data interface 1130, which is used for data transmission with the outside world.

[0245] For example, the device 1100 is used to implement the method 400 in the above embodiments.

[0246] For example, the device 1100 is used to achieve Figure 7 The method in the illustrated embodiment.

[0247] For example, the device 1100 is used to achieve Figure 8 Method 0 in the illustrated embodiment.

[0248] like Figure 12 As shown, this application embodiment also provides a neural network architecture search apparatus 1200. Apparatus 1200 can be used to execute the method 400 in the above embodiments. Apparatus 1200 includes a main controller 1210 for a convolutional neural network architecture search library, a convolutional neural network architecture generator 1220, a model converter 1230, and a model trainer 1240.

[0249] The main controller 1210 of the convolutional neural network structure search library is used to control the convolutional neural network structure generator 1220, the model converter 1230, and the model trainer 1240.

[0250] For example, the main controller 1210 of the convolutional neural network architecture search library is used to execute... Figure 4 Step S410 in the illustrated embodiment.

[0251] For example, the main controller 1210 of the convolutional neural network architecture search library is used to execute... Figure 6 Steps S410 and S420 in the illustrated embodiment.

[0252] The convolutional neural network structure generator 1220 is used to generate the convolutional neural network structure to be searched.

[0253] For example, the convolutional neural network structure generator 1220 is used to perform step S420 in the method 400 provided in the above method embodiment.

[0254] The model converter 1230 is used to convert the convolutional neural network structure generated by the convolutional neural network structure generator 1220 into a convolutional neural network model.

[0255] The model trainer 1240 is used to train the convolutional neural network model obtained by the model converter 1230 in order to obtain the evaluation index of the convolutional neural network model.

[0256] For example, the model converter 1230 and the model trainer 1240 are used as a whole in step S430 of the method 400 provided in the above method embodiment.

[0257] As an example, the schematic process of device 1200 performing automated convolutional neural network architecture search is as follows: Figure 13 As shown.

[0258] S1310, the main controller 1210 of the convolutional neural network structure search library passes the relevant parameters used to generate the convolutional neural network structure to the convolutional neural network structure generator 1220.

[0259] For example, the main controller 1210 of the convolutional neural network architecture search library is called by the script file, and the main controller 1210 of the convolutional neural network architecture search library receives the relevant parameters for generating the convolutional neural network architecture from the script file.

[0260] The parameters used to generate the convolutional neural network structure include, for example, the number of layers in the convolutional neural network, the total number of parameters in the convolutional neural network, the kernel size, the ratio of channels in the convolutional layers, and the options for the number of channels in the convolutional layers.

[0261] The description of the channel ratio of the convolutional layer can be found in the method embodiment above, and will not be repeated here.

[0262] S1320, the convolutional neural network structure generator 1220 generates a convolutional neural network structure (or, constructs a convolutional neural network structure) based on the relevant parameters passed by the main controller 1210 of the convolutional neural network structure search library, and passes the convolutional neural network structure to the model converter 1230.

[0263] For example, in the process of constructing a convolutional neural network structure each time, the convolutional neural network structure generator 1220 randomly selects the channel ratio of the convolutional layer, randomly selects the number of channels of the convolutional layer based on the value calculated based on the channel ratio, selects the kernel size of the convolutional layer, and randomly establishes cross-layer connections.

[0264] S1330, the model converter 1230 converts the convolutional neural network structure generated by the convolutional neural network structure generator 1220 into a convolutional neural network model, and passes the convolutional neural network model to the model trainer 1240.

[0265] S1340, the model trainer 1240 performs rapid training on the convolutional neural network model and feeds back the evaluation metrics obtained after training to the main controller 1210 of the convolutional neural network structure search library.

[0266] S1350, the main controller of the convolutional neural network structure search library, 1210 statistical evaluation indicators, and retains the convolutional neural network structures with the highest ranking of evaluation indicators and their corresponding convolutional neural network models.

[0267] For example, the main controller 1210 of the convolutional neural network architecture search library retains the top ten convolutional neural network architectures and their corresponding convolutional neural network models ranked by evaluation metrics.

[0268] S1360, the main controller 1210 of the convolutional neural network structure search library determines whether the total time for searching the convolutional neural network structure has been exhausted. If yes, proceed to step S1370; otherwise, proceed to step S1320.

[0269] For example, the total time spent searching for convolutional neural network architectures is 8 hours.

[0270] S1370, the main controller 1210 of the convolutional neural network architecture search library passes the convolutional neural network models ranked high by evaluation metrics to the model trainer 1240 for sufficient training, and re-ranks them based on the new evaluation metrics fed back by the model trainer 1240.

[0271] S1380, based on the evaluation metric ranking obtained in step S1370, the main controller 1210 of the convolutional neural network structure search library outputs the convolutional neural network structure with the highest evaluation metric ranking. The convolutional neural network structure output here can be considered the optimal convolutional neural network structure obtained through the convolutional neural network structure search.

[0272] At this point, device 1200 has completed the automated search for convolutional neural network structures.

[0273] like Figure 14 As shown in the figure, this application embodiment also provides a server device 1400. The server device 1400 includes a storage module 1410, a processing module 1420, and an output module 1430. The processing module 1420 is any one of the devices 1000, 1100, and 1200 provided in the above embodiments.

[0274] Storage module 1410 is used to store the training data required for searching the convolutional neural network architecture.

[0275] For example, the training data includes both a training set and a test set.

[0276] In machine learning, samples are generally divided into three parts: training set, validation set, and test set. The training set is used to build the model, and the test set is used to evaluate the performance of the ultimately selected optimal model.

[0277] The processing module 1420 is used to perform neural network structure search on the convolutional neural network using the training data stored in the storage module 1410.

[0278] The output module 1430 is used to output the convolutional neural network structure searched by the processing module 1420.

[0279] For example, embodiments of this application can be applied to a convolutional neural network architecture search system with 3x super-resolution. For example, storage module 1410 stores a dataset with 3x super-resolution, including, for example, a training set and a test set with 3x super-resolution. The convolutional neural network architecture output by output module 1430 is a convolutional neural network architecture applied to 3x super-resolution.

[0280] like Figure 15 As shown, this application embodiment also provides a neural network structure search system 1500, including a convolutional neural network structure search server 1510 and a convolutional neural network structure running platform 1520. The convolutional neural network structure search server 1510 is the server device 1400 provided in the above embodiment.

[0281] The convolutional neural network architecture search server 1510 is used to deploy the searched convolutional neural network architecture to the convolutional neural network architecture runtime platform 1520.

[0282] The Convolutional Neural Network Architecture Runtime Platform 1520 is used to run deployed convolutional neural network architectures.

[0283] For example, the convolutional neural network architecture runtime platform 1520 can be used as a terminal, or as another application platform for convolutional neural networks.

[0284] For example, the convolutional neural network architecture operating platform 1520 is a mobile phone.

[0285] For example, embodiments of this application can be applied to a convolutional neural network architecture search system with three times super-resolution, and super-resolution processing can be performed on the convolutional neural network architecture running platform 1520.

[0286] This application also provides a computer-readable medium storing program code for execution by a device, the program code including methods for performing the above embodiments.

[0287] This application also provides a computer program product containing instructions that, when run on a computer, cause the computer to perform the methods described in the above embodiments.

[0288] This application also provides a chip, which includes a processor and a data interface. The processor reads instructions stored in the memory through the data interface and executes the method described in the above embodiments.

[0289] Optionally, as one implementation, the chip may further include a memory storing instructions, and a processor for executing the instructions stored in the memory. When the instructions are executed, the processor is used to perform the methods described in the above embodiments.

[0290] Figure 16 A chip hardware structure provided in this application embodiment includes a neural network processor 1600. This chip can be disposed in any one or more of the following devices:

[0291] like Figure 10 The device 1000 shown, such as Figure 11 The device 1100 shown, such as Figure 12 The device 1200 shown, such as Figure 14 The server device 1400 shown is as follows: Figure 15 In the system 1500 shown.

[0292] Method 400 in the above method embodiments Figure 7 The method or in the illustrated embodiments Figure 8 The methods in the illustrated embodiments can all be implemented as follows: Figure 16 This is achieved in the chip shown.

[0293] The neural network processor 1600 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the neural network processor 1600 is the arithmetic circuit 1603, and the controller 1604 controls the arithmetic circuit 1603 to obtain data from the memory (weight memory 1602 or input memory 1601) and perform calculations.

[0294] In some implementations, the arithmetic circuit 1603 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 1603 is a two-dimensional pulsating array. The arithmetic circuit 1603 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 1603 is a general-purpose matrix processor.

[0295] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit 1603 retrieves the corresponding data for matrix B from the weight memory 1602 and caches it in each PE (Engineer Component) of the arithmetic circuit 1603. The arithmetic circuit 1603 retrieves the data for matrix A from the input memory 1601 and performs matrix operations with matrix B. The partial or final result of the obtained matrix is ​​stored in the accumulator 1608.

[0296] The vector computation unit 1607 can further process the output of the arithmetic circuit 1603, such as vector multiplication, vector addition, exponentiation, logarithmic operations, size comparisons, etc. For example, the vector computation unit 1607 can be used for network computation in non-convolutional / non-FC layers of neural networks, such as pooling, batch normalization, local response normalization, etc.

[0297] In some implementations, the vector computation unit 1607 can store the processed output vector into a unified memory (also called a unified buffer) 1606. For example, the vector computation unit 1607 can apply a nonlinear function to the output of the arithmetic circuit 1603, such as a vector of accumulated values, to generate activation values. In some implementations, the vector computation unit 1607 generates normalized values, merged values, or both. In some implementations, the processed output vector can be used as an activation input to the arithmetic circuit 1603, for example, for use in subsequent layers of a neural network.

[0298] Method 400 in the above method embodiments Figure 7 The method or in the illustrated embodiments Figure 8 The method in the illustrated embodiment can be performed by 1603 or 1607.

[0299] The unified memory 1606 is used to store input data and output data.

[0300] The direct memory access controller (DMAC) 1605 can be used to move input data from external memory to input memory 1601 and / or unified memory 1606, store weight data from external memory into weight memory 1602, and store data from unified memory 1606 into external memory.

[0301] The bus interface unit (BIU) 1610 is used to enable interaction between the main CPU, DMAC and instruction fetch memory 1609 via a bus.

[0302] The instruction fetch buffer 1609, which is connected to the controller 1604, is used to store the instructions used by the controller 1604.

[0303] The controller 1604 is used to call the instructions cached in the instruction memory 1609 to control the operation of the computing accelerator.

[0304] Generally, the unified memory 1606, input memory 1601, weight memory 1602, and instruction fetch memory 1609 are all on-chip memories, while the external memory is memory outside the NPU. This external memory can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or other readable and writable memory.

[0305] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the application.

[0306] It should be noted that the various numerical designations such as first, second, third, or fourth used in this document are merely for the convenience of description and are not intended to limit the scope of the embodiments of this application.

[0307] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0308] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0309] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0310] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0311] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0312] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: Universal Serial Bus flash disks (USB flash disks, UFDs) (UFDs can also be simply referred to as USB flash drives or USB flash drives), portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0313] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for searching neural network structures, characterized in that, Applied to a terminal, the method includes: The number of channels in each convolutional layer of a convolutional neural network is determined, wherein the proportion of the number of channels in any two consecutive convolutional layers is the same, and the convolutional neural network is applied to image processing tasks. The search space of the convolutional neural network is constructed based on the number of channels in each convolutional layer; Search for the neural network structure of the convolutional neural network based on its search space.

2. The method according to claim 1, characterized in that, The convolutional neural network includes a first convolutional layer and a second convolutional layer, wherein the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in the convolutional neural network, and the first convolutional layer is the convolutional layer preceding the second convolutional layer. Determining the number of channels in each convolutional layer of the convolutional neural network includes: A first value is randomly selected from a preset range as the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer; The number of channels in the first convolutional layer is determined as the second value based on the product of the first value and the number of channels in the second convolutional layer.

3. The method according to claim 2, characterized in that, Before constructing the search space of the convolutional neural network based on the number of channels of each convolutional layer, the method further includes: A third value is randomly selected near the second value, and the number of channels in the first convolutional layer is updated to the third value.

4. The method according to any one of claims 1 to 3, characterized in that, The step of constructing the search space of the convolutional neural network based on the number of channels in each convolutional layer includes: The search space of the convolutional neural network is constructed based on the number of channels in each convolutional layer, the number of layers in the convolutional neural network, and the size of the convolutional kernels in each convolutional layer.

5. The method according to any one of claims 1 to 3, characterized in that, The step of searching the neural network structure of the convolutional neural network based on its search space includes: Using the training data of the image processing task, the neural network structure of the convolutional neural network is searched according to the search space of the convolutional neural network.

6. A device for searching neural network structures, characterized in that, include: A determining unit is used to determine the number of channels in each convolutional layer of a convolutional neural network, wherein the proportion of the number of channels in any two consecutive convolutional layers is the same, and the convolutional neural network is applied to an image processing task. A construction unit is used to construct the search space of the convolutional neural network based on the number of channels in each convolutional layer; The search unit is used to search for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network.

7. The apparatus according to claim 6, characterized in that, The convolutional neural network includes a first convolutional layer and a second convolutional layer, wherein the first convolutional layer and the second convolutional layer are two consecutive convolutional layers in the convolutional neural network, and the first convolutional layer is the convolutional layer preceding the second convolutional layer. The determining unit is used for: A first value is randomly selected from a preset range as the ratio of the number of channels in the first convolutional layer to the number of channels in the second convolutional layer; The number of channels in the first convolutional layer is determined as the second value based on the product of the first value and the number of channels in the second convolutional layer.

8. The apparatus according to claim 7, characterized in that, The determining unit is further configured to, before the building unit constructs the search space of the convolutional neural network based on the number of channels of each convolutional layer, randomly select a third value near the second value and update the number of channels of the first convolutional layer to the third value.

9. The apparatus according to any one of claims 6 to 8, characterized in that, The construction unit is used to construct the search space of the convolutional neural network based on the number of channels in each convolutional layer, the number of layers in the convolutional neural network, and the size of the convolutional kernels in each convolutional layer.

10. The apparatus according to any one of claims 6 to 8, characterized in that, The search unit is used to search for the neural network structure of the convolutional neural network based on the search space of the convolutional neural network, using the training data of the image processing task.

11. A server device, characterized in that, include: The storage module is used to store the training data required for searching the convolutional neural network architecture; The apparatus for neural network structure search as described in any one of claims 6-10 is used to perform neural network structure search on a convolutional neural network using training data stored in the storage module, wherein the convolutional neural network is applied to an image processing task; The output module is used to output the convolutional neural network structure searched by the neural network structure search device.

12. A system for searching neural network structures, characterized in that, Includes a neural network architecture operating platform and the server device as described in claim 11; The server device is used to deploy the searched convolutional neural network structure to the neural network structure running platform, wherein the convolutional neural network corresponding to the convolutional neural network structure is applied to image processing tasks. The neural network architecture runtime platform is used to run the deployed convolutional neural network architecture.

13. A data processing apparatus, characterized in that, include: Memory, used to store executable instructions; A processor for invoking and executing the executable instructions in the memory to perform the method of any one of claims 1 to 5.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores program instructions that, when executed by a processor, implement the method of any one of claims 1 to 5.

15. A computer program product, characterized in that, The computer program product includes computer program code, which, when run on a computer, implements the method of any one of claims 1 to 5.

Images