A method, apparatus, server and system for training a neural network

Abstract

Embodiments of the present application disclose a method and device for training a neural network, a server and a system, which are used for reducing the interference between sub-networks in a super-network, so as to improve the performance of the sub-networks in the super-network. The method comprises the following steps: constructing a first super-network, wherein the first super-network comprises X sub-networks sharing parameters; training Y sub-networks in the X sub-networks, and updating the parameters of the first super-network based on the training results of the Y sub-networks; constructing a second super-network based on the parameters of the first super-network, wherein the second super-network comprises M branch networks, the M branch networks comprise the X sub-networks, any one of the X sub-networks is only included in one branch network, and all the sub-networks in the same branch network share parameters; training Z sub-networks in the X sub-networks, and then updating the parameters of the branch networks to which the Z sub-networks belong based on the training results of the Z sub-networks.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, device, server, and system for training a neural network. Background Art

[0002] With the development of artificial intelligence (AI) technology, neural networks are increasingly being used in terminal devices. Because different application scenarios and terminal devices have different requirements for neural networks, it is necessary to select the appropriate neural network based on the application scenario or terminal device. Previously, neural networks were manually designed. In recent years, the Neural Architecture Search (NAS) algorithm has emerged, which can automatically search for suitable neural networks.

[0003] The Once For All (OFA) algorithm is a key NAS algorithm. The OFA algorithm's neural network search process involves first selecting a target network, then designing rules to modify the target network to obtain a search space, and then constructing a supernetwork based on the search space. This supernetwork encompasses all subnetworks in the search space. After all subnetworks in the supernetwork have been trained to converge, a subset of subnetworks are sampled from the supernetwork for performance evaluation. Finally, based on the performance evaluation results, a subnetwork is selected as the neural network deployed on the terminal device.

[0004] Since all sub-networks in a super-network share parameters, during the training process of the sub-networks in the super-network, a change in the parameters of one sub-network will cause changes in the parameters of other sub-networks that share parameters with it. That is, the interference between the parameters of the sub-networks in the super-network is large, which has a greater impact on the performance of the sub-networks in the super-network. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, server, and system for training a neural network, which are used to reduce the interference of parameters between sub-networks in a super-network, thereby improving the performance of the sub-networks in the super-network.

[0006] A first aspect of an embodiment of the present application provides a method for training a neural network, comprising: constructing a first supernetwork based on a search space, wherein the first supernetwork includes X subnetworks, and the X subnetworks share parameters, wherein the X subnetworks share parameters, wherein for any two subnetworks in the X subnetworks, the parameters of the parts with the same network structure are the same, and X is an integer greater than 1; training Y subnetworks in the X subnetworks, wherein there may be multiple methods for training the Y subnetworks; updating the parameters of the first supernetwork based on the training results of the Y subnetworks in the X subnetworks, wherein Y is a positive integer less than or equal to X; and updating the parameters of the first supernetwork based on the parameters of the first supernetwork. A second supernetwork is constructed. Specifically, the X subnetworks in the first supernetwork can be divided into M groups. The constructed second supernetwork includes M branch networks, each of the M branch networks contains X subnetworks, and any subnetwork in the X subnetworks is only contained in one branch network. All subnetworks in the same branch network share parameters, where M is an integer greater than 1. Z subnetworks in the X subnetworks are trained, where multiple methods for training the Z subnetworks can be used. Based on the training results of the Z subnetworks in the X subnetworks, parameters of the branch networks to which each of the Z subnetworks belongs are updated, where Z is a positive integer less than or equal to X.

[0007] The method for selecting Y subnetworks from X subnetworks can be the same as the method for selecting Z subnetworks from X subnetworks. Taking the selection of Y subnetworks as an example, Y subnetworks can be selected from X subnetworks by changing the value of the elastic kernel size k in the first supernetwork; Y subnetworks can be selected from X subnetworks by changing the value of the elastic depth d in the first supernetwork; and Y subnetworks can be selected from X subnetworks by changing the value of the elastic width e in the first supernetwork.

[0008] First, Y subnetworks out of X subnetworks are trained, and the parameters of the first supernetwork are updated based on the training results of the Y subnetworks, thereby ensuring that the X subnetworks have good performance. Then, Z subnetworks out of X subnetworks are trained, and the parameters of the branch networks to which each of the Z subnetworks belongs are updated based on the training results of the Z subnetworks. Because all subnetworks in the same branch network share parameters, and based on the training results of one subnetwork out of Z subnetworks, only the parameters of the branch network to which that subnetwork belongs are updated, without causing changes to the parameters of subnetworks in other branch networks. This reduces mutual interference between the parameters of the X subnetworks, further improves the performance of the X subnetworks, and ultimately improves the performance of the neural network deployed on the terminal device.

[0009] In one implementation, P subnetworks are selected from X subnetworks as anchor networks based on preset conditions, the Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks are the subnetworks among the X subnetworks that meet the conditions, the Q non-anchor networks are the subnetworks other than the P anchor networks among the X subnetworks, and both P and Q are positive integers less than X.

[0010] Since the Y sub-networks include an anchor network selected based on conditions, the parameters of the first super-network are updated according to the training results of the Y sub-networks, so that the parameters of the first super-network can ensure that the anchor network and other anchor networks with similar network structures to the anchor network have better performance.

[0011] In one implementation, P subnetworks are selected from X subnetworks as anchor networks based on preset conditions, the Z subnetworks include a first subnetwork, a second subnetwork, at least one of the P anchor networks, and Q non-anchor networks, the first subnetwork is a subnetwork with the largest number of neurons among the X subnetworks, the second network is a subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks are subnetworks among the X subnetworks that meet the conditions, the Q non-anchor networks are subnetworks other than the P anchor networks among the X subnetworks, and both P and Q are positive integers less than X.

[0012] Since the Z sub-networks include an anchor network selected based on preset conditions, the parameters of the second super-network are updated according to the training results of the Z sub-networks, so that the parameters of the second super-network can ensure that the anchor network and other anchor networks with similar network structures to the anchor network have better performance.

[0013] In one implementation, after constructing the first supernetwork and before updating the parameters of the first supernetwork based on the training results of Y subnetworks among the X subnetworks, the method further includes: selecting P subnetworks that meet a condition from the X subnetworks as P anchor networks; wherein the condition can be set according to actual needs.

[0014] Select P sub-networks that meet the conditions from the X sub-networks as P anchor networks, so that Y sub-networks can be selected for training based on the P anchor networks.

[0015] In one implementation, constructing the first supernetwork includes: establishing a search space based on a target network, where the target network can be a second-version mobile network or a third-version mobile network. In addition, the target network can also be a neural network of other structures, and the search space indicates the range of variation of the network structure of the target network; and constructing the first supernetwork based on the search space.

[0016] This implementation provides a specific and feasible solution for constructing the first super network.

[0017] In one implementation, the target network includes an input layer, at least one stage and a head, each stage includes at least one block, each block includes a dimension-raising point convolution, a depth convolution and a dimension-reducing convolution; the variation range is related to at least one of the following parameters: a scaling ratio of the number of channels of the target network and a type of the head; in addition, the variation range may also be related to the elastic kernel size of each block in the at least one block, the elastic depth of each stage in the at least one stage, and the elastic width of each block in the at least one block; wherein the dimension-raising point convolution can be expressed as 1*1*(e·Cin), 1*1 represents the size of the convolution kernel of the dimension-raising point convolution, Here, Cin represents the number of input channels of the dimensionality-increasing point convolution, and e represents the ratio of the number of output channels of the dimensionality-increasing point convolution to the number of input channels. Depthwise convolution can be expressed as k*k*(e·Cin), where k*k represents the size of the convolution kernel of the depthwise convolution, Cin represents the number of input channels of the depthwise convolution, and e represents the ratio of the number of output channels of the depthwise convolution to the number of input channels. Dimensionality-reducing convolution can be expressed as 1*1*Cout, where 1*1 represents the size of the convolution kernel of the dimensionality-reducing point convolution, and Cout represents the number of output channels of the dimensionality-reducing convolution. The input layer can be a dimensionality-increasing point convolution. In addition, a layer of dimensionality-increasing point convolution can be included between at least one stage and the head.

[0018] The elastic kernel size is the convolution kernel size of the depthwise convolution in the block, that is, the elastic kernel size can be represented by k; the elastic depth is the number of blocks in the stage, and the embodiment of the present application uses d to represent the number of blocks in the stage; the elastic width is the ratio of the number of output channels of the dimensionality-increasing point convolution in the block to the number of input channels, that is, the elastic width can be represented by e; the scaling ratio of the number of channels of the target network indicates the ratio of the actual number of output channels of the input layer to the set number of output channels, the ratio of the actual number of output channels of the head to the set number of output channels, and the ratio of the actual number of output channels of the dimensionality-reducing convolution in each block to the set number of output channels of the dimensionality-reducing convolution.

[0019] The search space defines the range of variation of the network structure based on the target network, so that it can cover neural networks with various network structures; and in addition to defining the range of variation of the network structure from the three dimensions of elastic core size, elastic depth and elastic width, the embodiment of the present application also defines the range of variation of the network structure from the two dimensions of the scaling ratio of the number of channels and the type of head, thereby further expanding the coverage of the search space, so that the search space can cover neural networks with more network structures including ultra-lightweight neural networks.

[0020] In one implementation, the target network is a second-version mobile network or a third-version mobile network; when the target network is a second-version mobile network, the type of the header is the header of the second-version mobile network or the header of the third-version mobile network; when the target network is a third-version mobile network, the type of the header is the header of the second-version mobile network or the header of the third-version mobile network; the second-version mobile network includes five stages, and each stage in the second-version mobile network includes two blocks, three blocks, four blocks, three blocks, and three blocks in sequence from close to the input layer to away from the input layer; the third-version mobile network includes five stages, and each stage in the third-version mobile network includes two blocks, three blocks, four blocks, two blocks, and three blocks in sequence from close to the input layer to away from the input layer.

[0021] In this implementation method, when the target network is a second-version mobile network, the coverage of the search space is expanded through the header of the second-version mobile network and the header of the third-version mobile network; when the target network is a third-version mobile network, the coverage of the search space is expanded through the header of the second-version mobile network and the header of the third-version mobile network.

[0022] In one implementation, the range of variation is associated with multiple scaling ratios of the number of channels of the target network; the scaling ratio of the number of channels of the Y sub-networks includes at least two of the multiple scaling ratios and / or the scaling ratio of the number of channels of the Z sub-networks includes at least two of the multiple scaling ratios.

[0023] In this implementation, the scaling ratio of the number of channels of the Y sub-networks includes at least two of the multiple scaling ratios. Compared with the scaling ratio of the number of channels of the Y sub-networks including only one scaling ratio, this implementation enables the Y sub-networks to include sub-networks with more network structures, thereby improving the training effect of the first super network; similarly, the scaling ratio of the number of channels of the Z sub-networks includes at least two of the multiple scaling ratios. Compared with the scaling ratio of the number of channels of the Z sub-networks including only one scaling ratio, this implementation enables the Z sub-networks to include sub-networks with more network structures, thereby realizing training of sub-networks with more network structures, thereby improving the training effect of the first super network.

[0024] In one implementation, in the Y sub-networks, the scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same.

[0025] The scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same, thereby ensuring the adequacy of the training of the sub-networks in each scaling ratio.

[0026] In one implementation, in the Z sub-networks, the scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same.

[0027] The scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same, thereby ensuring the adequacy of the training of the sub-networks in each scaling ratio.

[0028] In one implementation, the Y subnetworks include a subnetwork with a header of a mobile terminal network of version 2, and a subnetwork with a header of a mobile terminal network of version 3; and / or the Z subnetworks include a subnetwork with a header of a mobile terminal network of version 2, and a subnetwork with a header of a mobile terminal network of version 3.

[0029] In this implementation, the Y sub-networks include a sub-network with a head of the mobile terminal network of the second version, and a sub-network with a head of the mobile terminal network of the third version, so that both the training of the sub-network including the head of the mobile terminal network of the second version and the training of the sub-network including the head of the mobile terminal network of the third version can be implemented; similarly, the Z sub-networks also include a sub-network with a head of the mobile terminal network of the second version, and a sub-network with a head of the mobile terminal network of the third version, so both the training of the sub-network including the head of the mobile terminal network of the second version and the training of the sub-network including the head of the mobile terminal network of the third version can be implemented.

[0030] In one implementation, among the Y sub-networks, the number of sub-networks including headers of the second version of the mobile terminal network is the same as the number of sub-networks including headers of the third version of the mobile terminal network.

[0031] In this implementation, the number of sub-networks including the head of the second version of the mobile terminal network is the same as the number of sub-networks including the head of the third version of the mobile terminal network, which can ensure the sufficiency of the training of the sub-network including the head of the second version of the mobile terminal network, and can also ensure the sufficiency of the training of the sub-network including the head of the third version of the mobile terminal network.

[0032] In one implementation, among the Z sub-networks, the number of sub-networks including headers of the second version of the mobile terminal network is the same as the number of sub-networks including headers of the third version of the mobile terminal network.

[0033] In this implementation, the number of sub-networks including the head of the second version of the mobile terminal network is the same as the number of sub-networks including the head of the third version of the mobile terminal network, which can ensure the sufficiency of the training of the sub-network including the head of the second version of the mobile terminal network, and can also ensure the sufficiency of the training of the sub-network including the head of the third version of the mobile terminal network.

[0034] In one implementation, the range of variation is related to multiple scaling ratios of the number of channels of the target network; the Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belong to the X subnetworks, and the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, the Q non-anchor networks are other subnetworks among the X subnetworks excluding the P anchor networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0035] In this implementation, the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, so that the P anchor networks include sub-networks of more network structures, thereby enabling training of sub-networks of more network structures, thereby improving the training effect of the first super network.

[0036] In one implementation, in the P anchor networks, the scaling ratio of the number of channels is such that the number of anchor networks in each scaling ratio in at least two scaling ratios is the same.

[0037] In this implementation, in the P anchor networks, the scaling ratio of the number of channels of the target network is the same as the number of anchor networks of at least two preset ratios, thereby ensuring the adequacy of training of the sub-networks of each scaling ratio.

[0038] In one implementation, the range of variation is related to multiple scaling ratios of the number of channels of the target network; the Z subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belong to the X subnetworks, and the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, the Q non-anchor networks are other subnetworks among the X subnetworks excluding the P anchor networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0039] In this implementation, the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, so that the P anchor networks include sub-networks of more network structures, thereby enabling training of sub-networks of more network structures, thereby improving the training effect of the first super network.

[0040] In one implementation, in the P anchor networks, the scaling ratio of the number of channels is such that the number of anchor networks in each scaling ratio in at least two scaling ratios is the same.

[0041] In this implementation, the scaling ratio of the number of channels of the target network in the P anchor networks is the same as the number of anchor networks of at least two preset ratios, thereby ensuring the adequacy of training of the sub-networks of each scaling ratio.

[0042] According to a second aspect of an embodiment of the present application, there is provided an apparatus for training a neural network, comprising: a construction unit for constructing a first supernetwork, wherein the first supernetwork includes X subnetworks, and the X subnetworks share parameters, where X is an integer greater than 1; a training unit for updating the parameters of the first supernetwork based on the training results of Y subnetworks out of the X subnetworks, where Y is a positive integer less than or equal to X; a construction unit for constructing a second supernetwork based on the parameters of the first supernetwork, wherein the second supernetwork includes M branch networks, wherein the M branch networks include X subnetworks, and any one of the X subnetworks is included in only one branch network, and all subnetworks in the same branch network share parameters, where M is an integer greater than 1; and a training unit for updating the parameters of the branch networks to which each of the Z subnetworks belongs based on the training results of Z subnetworks out of the X subnetworks, where Z is a positive integer less than or equal to X.

[0043] In one implementation, the Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks. The first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks are the subnetworks that meet the conditions among the X subnetworks, and the Q non-anchor networks are the subnetworks other than the P anchor networks among the X subnetworks. P and Q are both positive integers less than X.

[0044] In one implementation, the Z subnetworks include a first subnetwork, a second subnetwork, P anchor point networks, and at least one of Q non-anchor point networks. The first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor point networks are subnetworks that meet the conditions among the X subnetworks, and the Q non-anchor point networks are other subnetworks among the X subnetworks except the P anchor point networks. Both P and Q are positive integers less than X.

[0045] In one implementation, the training unit is further configured to select P sub-networks that meet the conditions from the X sub-networks as P anchor networks.

[0046] In one implementation, the construction unit is configured to establish a search space based on the target network, where the search space indicates a range of variation of the network structure of the target network; and construct a first super network based on the search space.

[0047] In one implementation, the target network includes an input layer, a head, and at least one stage located before the input layer and the head, each stage includes at least one block, and each block includes a dimension-increasing point convolution, a depth-wise convolution, and a dimension-reducing convolution; the range of variation is related to at least one of the following parameters: a scaling ratio of the number of channels of the target network and a type of head; the scaling ratio of the number of channels of the target network indicates at least one of the following ratios: a ratio of the actual number of output channels of the input layer to a set number of output channels, a ratio of the actual number of output channels of the head to a set number of output channels, and a ratio of the actual number of output channels of the dimension-reducing convolution in each block to a set number of output channels of the dimension-reducing convolution.

[0048] In one implementation, the target network is a second-version mobile network, and the type of the header is the header of the second-version mobile network or the header of the third-version mobile network; the second-version mobile network includes five stages, and each stage in the second-version mobile network includes two blocks, three blocks, four blocks, three blocks, and three blocks in sequence from close to the input layer to away from the input layer; the third-version mobile network includes five stages, and each stage in the third-version mobile network includes two blocks, three blocks, four blocks, two blocks, and three blocks in sequence from close to the input layer to away from the input layer.

[0049] In one implementation, the range of variation is associated with multiple scaling ratios of the number of channels of the target network; the scaling ratio of the number of channels of the Y sub-networks includes at least two of the multiple scaling ratios and / or the scaling ratio of the number of channels of the Z sub-networks includes at least two of the multiple scaling ratios.

[0050] In one implementation, the Y subnetworks include a subnetwork with a header of a mobile terminal network of version 2, and a subnetwork with a header of a mobile terminal network of version 3; and / or the Z subnetworks include a subnetwork with a header of a mobile terminal network of version 2, and a subnetwork with a header of a mobile terminal network of version 3.

[0051] In one implementation, the range of variation is related to multiple scaling ratios of the number of channels of the target network; the Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belong to the X subnetworks, and the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, the Q non-anchor networks are other subnetworks among the X subnetworks excluding the P anchor networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0052] In an implementation, the range of variation is related to a plurality of scaling ratios of the number of channels of the target network; the Z sub-networks include at least one of a first sub-network, a second sub-network, P anchor point networks, and Q non-anchor point networks, the first sub-network is a sub-network with the largest number of neurons in the X sub-networks, the second sub-network is a sub-network with the smallest number of neurons in the X sub-networks, the P anchor point networks belong to the X sub-networks, and the scaling ratios of the number of channels of the P anchor point networks include at least two of the plurality of scaling ratios, the Q non-anchor point networks are sub-networks other than the P anchor point networks in the X sub-networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0053] The specific implementation of each unit above, the related description, and the technical effects can be referred to the description of the first aspect of the embodiments of the present application.

[0054] The third aspect of the embodiments of the present application provides a server, including at least one processor and a memory, the memory stores computer execution instructions executable on the processor, when the computer execution instructions are executed by the processor, the server executes the method for training a neural network as described in any one of the embodiments of the first aspect.

[0055] The fourth aspect of the embodiments of the present application provides a chip or a chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, the at least one processor is used to run computer programs or instructions to execute the method for training a neural network as described in any one of the embodiments of the first aspect.

[0056] The fifth aspect of the embodiments of the present application provides a computer storage medium for storing computer software instructions for the server, which includes programs designed for the server;

[0057] The server can be the device for training a neural network as described in the second aspect.

[0058] The sixth aspect of the embodiments of the present application provides a computer program product, which includes computer software instructions, the computer software instructions can be loaded by a processor to implement the method for training a neural network as described in any one of the embodiments of the first aspect.

[0059] The seventh aspect of the embodiments of the present application provides a model deployment system, including a terminal device and a server.

[0060] The server is used to execute the method for training a neural network as described in any one of the embodiments of the first aspect, and is used to select a sub-network from the second super network to be deployed in the terminal device.

[0061] From the above technical solutions, the embodiments of the present application have the following advantages:

[0062] First, a first super network including X sub-networks is constructed, the X sub-networks share parameters, then Y sub-networks in the X sub-networks are trained, and the parameters of the first super network are updated based on the training results of the Y sub-networks, so as to ensure that the X sub-networks have good performance; thereafter, a second super network including M branch networks is constructed, each branch network includes at least one of the X sub-networks, then Z sub-networks in the X sub-networks are trained, and the parameters of the branch network to which the Z sub-networks belong are updated based on the training results of the Z sub-networks; since all sub-networks in the same branch network share parameters, and based on the training result of a sub-network in the Z sub-networks, only the parameters of the branch network to which the sub-network belongs are updated, without causing the parameters of the sub-networks in other branch networks to change, thereby reducing the mutual interference of the parameters among the X sub-networks, and further improving the performance of the X sub-networks, so that the performance of the neural network finally deployed on the terminal device is good. BRIEF DESCRIPTION OF DRAWINGS

[0063] Figure 1 A structural schematic diagram of an artificial intelligence main body framework;

[0064] Figure 2 A framework schematic diagram of an artificial intelligence system;

[0065] Figure 3 A schematic diagram of adaptively generating a neural network;

[0066] Figure 4 A schematic diagram of searching a neural network by using a NAS algorithm;

[0067] Figure 5 A schematic diagram of searching a neural network by using an OFA algorithm;

[0068] Figure 6 A schematic diagram of selecting a neural network for a terminal device by using the method for training a neural network provided by the embodiments of the present application;

[0069] Figure 7 A schematic diagram of the method for training a neural network in the embodiments of the present application;

[0070] Figure 8 A structural schematic diagram of the first embodiment of the first super network in the embodiments of the present application;

[0071] Figure 9 A schematic diagram of the Y sub-networks sharing an elastic kernel size in the embodiments of the present application;

[0072] Figure 10This is a structural diagram of the second embodiment of the first super network in the embodiments of the present application;

[0073] Figure 11 This is a schematic diagram of the structure of the first embodiment of the sub-network in the embodiments of the present application;

[0074] Figure 12 This is a schematic diagram of the structure of the second embodiment of the sub-network in the embodiments of the present application;

[0075] Figure 13 This is a schematic structural diagram of a third embodiment of a sub-network in an embodiment of the present application;

[0076] Figure 14 In the embodiments of this application, Figure 11 Schematic diagram of the position of the sub-network in the second super-network;

[0077] Figure 15 In the embodiments of this application, Figure 12 Schematic diagram of the position of the sub-network in the second super-network;

[0078] Figure 16 In the embodiments of this application, Figure 13 Schematic diagram of the position of the sub-network in the second super-network;

[0079] Figure 17 This is a schematic diagram of a first embodiment of constructing a second super network in an embodiment of the present application;

[0080] Figure 18 Schematic diagram of a device for training a neural network in an embodiment of the present application;

[0081] Figure 19 This is a schematic diagram of the structure of the server in the embodiment of the present application;

[0082] Figure 20 This is a structural diagram of the model deployment system in an embodiment of the present application;

[0083] Figure 21 This is a schematic diagram of the structure of the terminal device in the embodiment of the present application. DETAILED DESCRIPTION

[0084] The following describes the embodiments of the present invention in conjunction with the accompanying drawings. The terms used in the embodiments of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0085] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0086] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, and this is merely a way of distinguishing the objects of the same attributes when describing them in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0087] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The following diagram illustrates a structural diagram of the AI ​​framework. This framework is explained below from two perspectives: the "intelligent information chain" (horizontal axis) and the "IT value chain" (vertical axis). The "intelligent information chain" reflects the entire process from data acquisition to processing. For example, it encompasses the general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, and intelligent execution and output. Throughout this process, data undergoes a condensed progression from "data-information-knowledge-wisdom." The "IT value chain," encompassing the entire process from the underlying infrastructure of human intelligence, information (provided and processed by technology), to the system's industrial ecosystem, reflects the value that AI brings to the information technology industry.

[0088] (1) Infrastructure

[0089] Infrastructure provides computing power for AI systems, enabling communication with the outside world and supporting this through a foundational platform. External communication occurs through sensors; computing power is provided by intelligent chips (CPUs, NPUs, GPUs, ASICs, FPGAs, and other hardware accelerators). The foundational platform includes a distributed computing framework and network-related platform guarantees and support, including cloud storage and computing, and interconnected networks. For example, sensors communicate with the outside world to acquire data, which is then fed into the intelligent chips within the distributed computing system provided by the foundational platform for computation.

[0090] (2) Data

[0091] Data above the infrastructure layer represents data sources for AI. This data includes graphics, images, voice, and text, as well as IoT data from traditional devices. This includes business data from existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity.

[0092] (3) Data processing

[0093] Data processing generally includes data training, machine learning, deep learning, search, reasoning, decision-making, etc.

[0094] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0095] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0096] Decision-making refers to the process of making decisions after intelligent information is reasoned, and usually provides functions such as classification, sorting, and prediction.

[0097] (4) General ability

[0098] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0099] (5) Smart products and industry applications

[0100] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0101] The following introduces the application scenarios of the embodiments of the present application.

[0102] like Figure 2 As shown, the embodiment of the present application can be applied to Figure 2 The artificial intelligence system shown in FIG. The artificial intelligence system includes a terminal device and a server, and the terminal device and the server are connected via a communication link.

[0103] Among them, the terminal device can be any computing device, such as a personal computer, computer workstation, smartphone, tablet computer, smart camera, virtual reality (VR) device, smart car or other type of cellular phone, media consumption device, wearable device, set-top box, game console, etc.

[0104] Based on the advantages of neural networks, Figure 2The terminal device shown here deploys a neural network, which can be used in a variety of scenarios. For example, the terminal device can be used for identity verification using facial recognition technology; as another example, the terminal device can be used for augmented reality in games; as another example, the terminal device can be used to synthesize images taken by the user.

[0105] However, different application scenarios place different demands on neural networks. For example, when a terminal device is used for identity authentication through facial recognition technology, a highly accurate neural network is required; when a terminal device is used for augmented reality in gaming, a faster-response neural network is required.

[0106] Furthermore, different types of terminal devices have different requirements for neural networks. For example, due to differences in underlying hardware, the computational latency of the same neural network can vary across different terminal devices. This difference can refer to different types of terminal devices, such as smartphones and tablets, or different models of the same type of terminal device, such as the Huawei P30 and Huawei P40.

[0107] Therefore, a suitable neural network can be selected for a specific terminal device based on the type of terminal device, application scenario, and other possible constraints.

[0108] In an embodiment of the present application, the server selects an appropriate neural network for the terminal device using Automatic Machine Learning (AutoML) technology. Automatic machine learning refers to using technologies such as transfer learning, automatic hyperparameter tuning, and Neural Architecture Search (NAS) to help users obtain a neural network that meets their needs, reduce repeated experiments, and improve modeling efficiency. The following describes in detail the process of the server selecting a neural network using automatic machine learning technology.

[0109] Specifically, if Figure 3 As shown, the user selects the type of task to be performed on the terminal device based on the application scenario. The task type can be image classification, detection, or natural language processing (NLP). For example, in one scenario, the user needs to identify the category of a group of animal pictures, so the task type can be image classification; in another scenario, the user needs to translate a paragraph of text, so the task type can be natural language understanding. The user selects the type of terminal device and the performance constraints on the neural network on the terminal device. The performance constraints can be the computational latency of the neural network, the size of the neural network, and the accuracy of the neural network.

[0110] After that, the server deployed with the AutoML system selects the basic network according to the task type. The embodiment of the present application does not specifically limit the type of the basic network. For example, the basic network can be the second version of the mobile terminal network (Efficient convolutional neural networks for mobile vision version 2, Mobilenet v2) or the third version of the mobile terminal network (Efficient convolutional neural networks for mobile vision version 3, Mobilenet v3). The server also determines the delay calculation method of different neural networks based on the type of terminal device.

[0111] Next, the server deployed with the AutoML system adaptively generates a suitable neural network for the terminal device based on the selected basic network, the determined delay calculation method, and the performance constraints of the neural network selected by the user.

[0112] Afterwards, the server deployed with the AutoML system uses transfer learning to train the selected neural network on the dataset, and uses a quantization algorithm to quantize the trained neural network to a size suitable for deployment on the terminal device. The quantization algorithm is a method of compressing the neural network, which approximates the parameters (including weights and biases) of the neural network represented by floating-point numbers (high precision) using low-precision integers. After quantization to low precision, optimization techniques on the mobile platform can be applied to accelerate the calculation process of the neural network. After quantization, the original neural network model capacity will also be reduced, making it better adapted to terminal devices with limited storage space.

[0113] It is understandable that the server can use a variety of methods to adaptively generate a suitable neural network for the terminal device. At present, the AutoML system mainly uses the Neural Architecture Search (NAS) algorithm, which can efficiently search for a large number of neural networks with high accuracy, low latency and a small number of parameters. Figure 4 The NAS algorithm is described.

[0114] like Figure 4 As shown in Figure 2, the process of searching neural networks using the NAS algorithm mainly includes:

[0115] Construct a search space. Specifically, based on a base network, design some network structure change rules for that base network. Based on these network structure change rules, the range of network structure changes for the base network can be determined. The search space indicates the range of network structure changes for the base network, and therefore, the search space can cover a variety of neural networks. For ease of explanation, the networks covered by the search space are referred to as subnetworks below.

[0116] Searching for subnetworks. Specifically, a certain number of subnetworks are searched from the search space based on a specific search strategy. It is understandable that since the search space covers a large number of subnetworks, different search strategies can search for different subnetworks.

[0117] Performance evaluation. Specifically, the performance of the searched subnetworks is evaluated based on a specific evaluation method, and the subnetwork with the best performance evaluation result is ultimately selected as the neural network of the terminal device. This evaluation method can evaluate the performance of the searched subnetwork based on multiple dimensions, such as the neural network's computational latency, size, and accuracy.

[0118] Based on the above description of the process of searching for neural networks using the NAS algorithm, it can be seen that the larger the search space, the more neural networks are covered, and therefore it is possible to search for a neural network that is more suitable for the terminal device; however, the larger the search space, the greater the search difficulty, and it will also lead to longer search times, more resource consumption, and difficulty in quickly responding to user needs.

[0119] To this end, the Once For All (Once For All, OFA) algorithm can be used to search for a suitable neural network for the terminal device. Among them, the OFA algorithm is one of the NAS algorithms. Figure 5 The process of searching neural networks using the OFA algorithm is explained.

[0120] Specifically, if Figure 5 As shown in Figure 2, the process of searching a neural network using the OFA algorithm mainly includes:

[0121] First, a search space is constructed. A supernetwork is then constructed based on the search space. This supernetwork encompasses all subnetworks in the search space. The supernetwork is then trained until all subnetworks in the supernetwork converge. A subset of subnetworks is sampled from the supernetwork for performance evaluation, and a structure-performance lookup table is constructed based on the performance evaluation results. This structure-performance lookup table records all sampled subnetworks and the corresponding performance of each sampled subnetwork.

[0122] Among them, the hypernetwork can be understood as: including at least one layer of space, each layer of space contains multiple substructures, each substructure includes at least one layer of neurons; two substructures in the multiple substructures can contain the same layer of neurons, that is, two substructures in the multiple substructures can contain the same structure.

[0123] In the embodiments of the present application, the hypernetwork can also be called a neural network.

[0124] The subnetwork can be understood as: selecting at least one substructure from each layer of the hypernetwork space, and then connecting the selected substructures in sequence according to the order of the spatial layers to form a neural network.

[0125] Since two substructures from the plurality of substructures can contain the same structure, two subnetworks selected from the supernetwork can contain the same structure.

[0126] In this way, after searching a certain number of sub-networks from the search space based on a specific search strategy, the server can query the structure performance lookup table to obtain the performance data of the searched sub-network, thereby searching for a suitable neural network for the terminal device.

[0127] Based on the relevant description of the process of searching neural networks using the OFA algorithm, it can be seen that the OFA algorithm decouples the neural network training process and the neural network search process. In this way, the super network can be trained before searching the neural network based on the search strategy to build a structural performance lookup table, thereby improving the efficiency and scalability of searching neural networks for terminal devices.

[0128] It should be noted that all sub-networks in a super-network share parameters. Sharing parameters in sub-networks means that if two sub-networks have identical network structures, the parameter types and values ​​of the identical network structures are the same, including weights and biases.

[0129] Because all subnetworks in a supernetwork share parameters, during the training of a subnetwork within the supernetwork, a change in the parameters of one subnetwork will cause changes in the parameters of all other subnetworks that share parameters with it. This results in significant interference between the parameters of the subnetworks within the supernetwork, significantly impacting the performance of the subnetworks within the supernetwork. Therefore, embodiments of the present application provide a method for training a neural network that reduces the interference between the parameters of the subnetworks within the supernetwork, thereby improving the performance of the subnetworks within the supernetwork.

[0130] The process of selecting a neural network for a terminal device using the method for training a neural network provided in an embodiment of the present application is as follows: Figure 6Specifically, on the backend, the server determines the target network based on the task type, constructs a search space based on the target network, then constructs a supernetwork based on the search space and trains the supernetwork. The subnetworks sampled from the supernetwork are then evaluated. Because different types of terminal devices require different performance calculation methods for neural networks, the server constructs a structural performance table based on the performance evaluation results of the self-network and the type of terminal device.

[0131] At the front end, the server searches for a sub-network based on the performance constraints and specific search strategies selected by the user, and then queries the performance of the sub-network through the structural performance table. Based on the query results, a sub-network is selected as the neural network deployed on the terminal device. The neural network is then transferred and learned using the data set, and the transferred neural network is quantized to a size suitable for the terminal device. Finally, the quantized neural network is deployed on the terminal device.

[0132] In the embodiment of the present application, the back end can be understood as offline, and the front end can be understood as online.

[0133] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0134] (1) Neural Network

[0135] A neural network can be composed of neural units. A neural unit can refer to an operation unit with xs and intercept 1 as input. The output of the operation unit can be:

[0136]

[0137] Where s = 1, 2, ..., n, where n is a natural number greater than 1, Ws is the weight of Xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolutional layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field. The local receptive field can be an area composed of several neural units.

[0138] (2) Deep Neural Networks

[0139] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. The neural networks inside a DNN can be divided into three categories according to the positions of different layers: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the layers in between are hidden layers. The layers are fully connected, that is, any neuron in the ith layer is connected to any neuron in the (i+1)th layer.

[0140] Although a DNN looks very complex, it is actually not complex in terms of the work of each layer. In simple terms, it is expressed by the following linear relationship expression: wherein, is an input vector, is an output vector, is a bias vector, W is a weight matrix (also called a coefficient), and a() is an activation function. Each layer only performs the following simple operation on the input vector to obtain the output vector Due to the large number of layers in a DNN, the number of coefficients W and bias vectors is also relatively large. These parameters in a DNN are defined as follows: taking the coefficient W as an example: assuming that in a three-layer DNN, the linear coefficient of the fourth neuron in the second layer to the second neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, and the subscripts correspond to the output third layer index 2 and the input second layer index 4.

[0141] In summary, the coefficient of the Kth neuron in the (L-1)th layer to the jth neuron in the Lth layer is defined as

[0142] It should be noted that the input layer has no W parameter. In a deep neural network, more hidden layers enable the network to better depict complex situations in the real world. In theory, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is a process of learning the weight matrix, and the ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by the vectors W of many layers).

[0143] (3) Convolutional neural network

[0144] A convolutional neural network (CNN) is a deep neural network with a convolutional architecture. It consists of a feature extractor consisting of convolutional layers and subsampling layers, which can be viewed as a filter. A convolutional layer is a layer of neurons that performs convolution processing on the input signal. In a convolutional layer of a CNN, a neuron can only connect to a subset of neurons in adjacent layers. A convolutional layer typically contains several feature planes, each of which can be composed of a rectangular arrangement of neurons. Neurons in the same feature plane share weights, which are referred to as convolution kernels. Shared weights can be understood as extracting image information in a position-independent manner. Convolution kernels can be initialized as matrices of random size, and during CNN training, they can learn to acquire reasonable weights. Furthermore, shared weights have the direct benefit of reducing the number of connections between layers of the CNN, thereby reducing the risk of overfitting.

[0145] (4) Loss function

[0146] During the training of a deep neural network, because we want the output of the deep neural network to be as close as possible to the desired predicted value, we can compare the current network's predicted value with the desired target value and then update the weight vector of each layer of the neural network based on the difference between the two. (Of course, there is usually an initialization process before the first update, which is to pre-configure the parameters for each layer in the deep neural network.) For example, if the network's predicted value is too high, the weight vector is adjusted to make it predict a lower value. This adjustment is continued until the deep neural network can predict the desired target value or a value very close to the desired target value. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value." This is the loss function (or objective function), which is an important equation used to measure the difference between the predicted value and the target value. For example, the loss function output value (loss) indicates a greater difference, so training a deep neural network becomes a process of minimizing this loss as much as possible.

[0147] (5) Backpropagation algorithm

[0148] Neural networks can use the back propagation (BP) algorithm to correct the initial neural network parameters during training, reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates error loss. This error loss is then backpropagated to update the initial neural network parameters, thereby converging the error loss. The BP algorithm is a backpropagation algorithm driven by error loss, aiming to optimize the neural network parameters, such as the weight matrix.

[0149] Among them, the neural network in the embodiments of this application mainly refers to a deep neural network.

[0150] The following combination Figure 7 The method for training a neural network provided in the embodiment of the present application is described. Figure 7 The present application provides a method for training a neural network, including:

[0151] Step 101: construct a first super network, wherein the first super network includes X sub-networks, and the X sub-networks share parameters, where X is an integer greater than 1.

[0152] The value of X can be set according to actual needs and is not specifically limited in the embodiments of the present application.

[0153] Based on the foregoing description, it can be seen that the shared parameters of the X sub-networks mean that for any two sub-networks among the X sub-networks, the parameters of the parts with the same network structure are the same; in the embodiment of the present application, the parameters include the weights and biases of the neural network, which will not be repeated below.

[0154] It should be noted that there are many methods for constructing the first super network, which are not specifically limited in the embodiments of the present application. As an example, the first super network including X sub-networks can be directly constructed.

[0155] As another example, constructing the first super network includes: first establishing a search space based on the target network, where the search space indicates a range of changes in the network structure of the target network, and then constructing the first super network based on the search space.

[0156] The target network may be the basic network mentioned above, and the embodiment of the present application does not specifically limit the type of the target network.

[0157] Exemplarily, the target network includes an input layer, at least one stage and a head, each stage includes at least one block, and each block includes a dimension-increasing point convolution, a depth-wise convolution and a dimension-reducing convolution.

[0158] Among them, the dimensionality-raising point convolution can be expressed as 1*1*(e·Cin), 1*1 represents the size of the convolution kernel of the dimensionality-raising point convolution, Cin here represents the number of input channels of the dimensionality-raising point convolution, and e here represents the ratio of the number of output channels of the dimensionality-raising point convolution to the number of input channels; the depth convolution can be expressed as k*k*(e·Cin), k*k represents the size of the convolution kernel of the depth convolution, Cin here represents the number of input channels of the depth convolution, and e here represents the ratio of the number of output channels of the depth convolution to the number of input channels; the dimensionality-reducing convolution can be expressed as 1*1*Cout, 1*1 represents the size of the convolution kernel of the dimensionality-reducing point convolution, and Cout here represents the number of output channels of the dimensionality-reducing convolution; the input layer can be the dimensionality-raising point convolution.

[0159] In addition, a layer of dimensionality-increasing point convolution can be included between at least one stage and the head.

[0160] It should be noted that in the same stage, the number of output channels of different blocks is the same, and the number of output channels of a block refers to the number of output channels of the dimensionality reduction convolution in the block.

[0161] Specifically, the target network can be a version 2 mobile network or a version 3 mobile network. When the target network is a version 3 mobile network, it includes an input layer, 5 stages, and the head of the version 3 mobile network. The 5 stages include 2, 3, 4, 2, and 3 blocks, respectively. In these 14 blocks, the values ​​of e are 4, 3, 3, 3, 3, 6, 2.5, 2.3, 2.3, 6, 6, 6, 6, and 6, respectively, and the values ​​of k are 3, 3, 5, 5, 5, 3, 3, 3, 3, 3, 3, 5, 5, and 5, respectively.

[0162] Based on the aforementioned target network, the range of variation is related to at least one of the following parameters: a scaling ratio of the number of channels of the target network and a type of head; in addition, the range of variation can also be related to the elastic kernel size of each block in at least one block, the elastic depth of each stage in at least one stage, and the elastic width of each block in at least one block.

[0163] It should be noted that at least one means that the range of variation can be determined based on the elastic core size of each block in at least one block, can be determined based on the elastic depth of each stage in at least one stage, can be determined based on the elastic width of each block in at least one block, can also be determined based on the scaling ratio of the number of channels of the target network, and can also be determined based on the type of head; in addition, the range of variation can also be determined based on a combination of two or more of the above.

[0164] Among them, the elastic kernel size is the convolution kernel size of the depth convolution in the block, that is, the elastic kernel size can be represented by k; the elastic depth is the number of blocks in the stage, and the embodiment of the present application uses d to represent the number of blocks in the stage; the elastic width is the ratio of the number of output channels of the dimensionality-raising point convolution in the block to the number of input channels, that is, the elastic width can be represented by e.

[0165] The scaling ratio of the number of channels of the target network indicates at least one of the following ratios: the ratio of the actual number of output channels of the input layer to the set number of output channels, the ratio of the actual number of output channels of the head to the set number of output channels, and the ratio of the actual number of output channels of the dimensionality reduction convolution in each block to the set number of output channels of the dimensionality reduction convolution. That is, the scaling ratio of the number of channels of the target network is equal to the ratio of the actual number of output channels of the input layer to the set number of output channels, and the ratio of the actual number of output channels of the head to the set number of output channels, wherein the set number of output channels of the dimensionality reduction convolution in each block is the number Cout of output channels of the dimensionality reduction convolution of the block in the target network. For ease of explanation, the scaling ratio of the number of channels of the target network is represented by scale below.

[0166] For ease of understanding, the scaling ratio of the number of channels of the target network is further explained below by taking any one block in at least one block (referred to herein as the target block) as an example.

[0167] Specifically, in the target network, the number of output channels Cout of the dimensionality reduction convolution in the target block is 4, that is, the set number of output channels of the dimensionality reduction convolution in the target block is 4. Based on this, the range of variation of the network structure of the target network can be determined by the scaling ratio scale of the number of channels of the target network. For example, when the scaling ratio of the number of channels of the target network is 1, the ratio of the actual number of output channels of the reduced-dimensionality convolution in the target block to the set number of output channels of the reduced-dimensionality convolution is 1, and the actual number of output channels of the reduced-dimensionality convolution in the target block is 4; when the scaling ratio of the number of channels of the target network is 0.75, the ratio of the actual number of output channels of the reduced-dimensionality convolution in the target block to the set number of output channels of the reduced-dimensionality convolution is 0.75, and the actual number of output channels of the reduced-dimensionality convolution in the target block is 3; when the scaling ratio of the number of channels of the target network is 0.5, the ratio of the actual number of output channels of the reduced-dimensionality convolution in the target block to the set number of output channels of the reduced-dimensionality convolution is 0.5, and the actual number of output channels of the reduced-dimensionality convolution in the target block is 2; when the scaling ratio of the number of channels of the target network is 0.25, the ratio of the actual number of output channels of the reduced-dimensionality convolution in the target block to the set number of output channels of the reduced-dimensionality convolution is 0.25, and the actual number of output channels of the reduced-dimensionality convolution in the target block is 1.

[0168] The actual number of output channels of the input layer and the actual number of output channels of the head change with the scale of the number of channels of the target network, which is the same as the change in the actual number of output channels of the dimensionality reduction convolution in the target block with the scale of the number of channels of the target network. Please refer to the above for details.

[0169] It can be understood that when k, d, e, and scale take different values, the range of change of the network structure of the target network in the search space is different.

[0170] Based on the above description, it can be known that the target network can be a mobile terminal network of version 2 or a mobile terminal network of version 3. For example, when the target network is a mobile terminal network of version 2, the type of header can include a header of a mobile terminal network of version 2 or a header of a mobile terminal network of version 3. When the target network is a mobile terminal network of version 3, the type of header can also include a header of a mobile terminal network of version 2 or a header of a mobile terminal network of version 3.

[0171] The second version of the mobile network includes five stages, and each stage in the second version of the mobile network includes two blocks, three blocks, four blocks, three blocks, and three blocks in sequence from close to the input layer to far away from the input layer; the third version of the mobile network includes five stages, and each stage in the third version of the mobile network includes two blocks, three blocks, four blocks, two blocks, and three blocks in sequence from close to the input layer to far away from the input layer.

[0172] Among them, the second version of the mobile terminal network is different from the third version of the mobile terminal network in terms of the input layer and the head, and all the blocks contained in the five stages include both the same blocks and different blocks; since the second version of the mobile terminal network and the third version of the mobile terminal network are existing neural networks, the embodiments of this application will not be described in detail here.

[0173] In the embodiments of the present application, the values ​​of k, d, e, and scale, as well as the type of header, can be changed according to actual needs, thereby changing the coverage of the search space. For example, if the target network is a third-version mobile network, the number of blocks d in each stage can be 1 to 4; in each block in each stage, the value of k can be any one of 3, 5, and 7, and the value of e can be any one of 3, 4, and 6; scale can be any one of 0.25, 0.5, 0.75, and 1; and the type of header can be a header for a second-version mobile network or a header for a third-version mobile network.

[0174] In an embodiment of the present application, the search space defines the range of variation of the network structure on the basis of the target network, thereby covering neural networks of various network structures; and, in addition to defining the range of variation of the network structure from three dimensions of elastic core size, elastic depth, and elastic width, the embodiment of the present application also defines the range of variation of the network structure from two dimensions of the scaling ratio of the number of channels of the target network and the type of head, thereby further expanding the coverage of the search space, so that the search space can cover neural networks of more network structures including ultra-lightweight neural networks.

[0175] After constructing the search space, the first super network can be constructed based on the variation range of the network structure of the target network. Figure 8 The first supernetwork will be described.

[0176] like Figure 8 As shown in , the search space corresponding to the first super network is constructed with the third version of the mobile terminal network as the target network. Figure 8 As can be seen in the figure, the first super network includes the input layer, stage 0, stage 1, stage 2, stage 3, stage 4, dimensionality-increasing point convolution, the head of the second version of the mobile terminal network, and the head of the third version of the mobile terminal network. The structural variation range of any stage in stage 0, stage 1, stage 2, stage 3, and stage 4 is the same. Taking stage 1 as an example, stage 1 includes blocks 1, 2 to block d. The structural variation range of each block in stage 1 is the same. Taking block 2 as an example, block 2 includes dimensionality-increasing point convolution 1*1*(e·Cin), depth convolution k*k*(e·Cin), and dimensionality-reducing convolution 1*1*Cout. The input layer, stage 0, stage 1, stage 2, stage 3, stage 4, and dimensionality-increasing point convolution are connected in sequence. The dimensionality-increasing point convolution can be connected to the head of the second version of the mobile terminal network or the head of the third version of the mobile terminal network.

[0177] for Figure 8 In the first supernetwork shown, when the values ​​of k, d, e, and scale are different, the structure of the first supernetwork is different. It can be seen that the first supernetwork includes multiple subnetworks.

[0178] In an embodiment of the present application, the range of variation of the network structure of the target network in the search space is determined based on at least one of the elastic core size of each block, the elastic depth in each stage, the elastic width of each block, the scaling ratio of the number of channels of the target network, and the type of head, so the search space can include the above five dimensions, thereby covering more neural networks to achieve coverage of ultra-lightweight neural networks.

[0179] It is understandable that after the first super network is constructed, corresponding initial parameters may be set for the first super network so as to train the first super network using the initial parameters.

[0180] Step 102: Update the parameters of the first super network based on the training results of Y sub-networks among the X sub-networks, where Y is a positive integer less than or equal to X.

[0181] Before performing step 102, Y sub-networks may be selected from the X sub-networks for training. It should be noted that various methods may be used to select Y sub-networks from the X sub-networks for training, and this embodiment of the present application does not specifically limit this. Different methods for selecting Y sub-networks for training may correspond to different types of Y sub-networks, and the parameters for updating the first super-network may also be different.

[0182] The following describes the method for selecting Y sub-networks for training.

[0183] First method:

[0184] After constructing the first supernetwork, and before updating the parameters of the first supernetwork based on the training results of Y subnetworks among the X subnetworks, P subnetworks that meet the conditions are selected from the X subnetworks as P anchor networks, and the subnetworks other than the P anchor networks among the X subnetworks are Q non-anchor networks. Before training, the process of selecting Y subnetworks from the X subnetworks for training by the following method includes: selecting all anchor networks from the P anchor networks, randomly selecting at least one non-anchor network from the Q non-anchor networks, and then selecting the subnetwork with the largest number of neurons and the subnetwork with the smallest number of neurons from the X subnetworks, thereby obtaining Y subnetworks; wherein P and Q are both positive integers less than X.

[0185] It should be noted that the embodiments of the present application do not specifically limit the conditions and can be set according to actual needs. For example, the conditions can be related to the values ​​of k, d, e, and scale. As an example, the conditions include: the value of scale is any one of 0.25, 0.5, 0.75, and 1, the value of d is any one of 2 and 4, the value of e is any one of 3 and 6, and the value of k is any one of 3 and 7. Based on this condition, P subnetworks can be selected from X subnetworks as anchor networks.

[0186] It's understandable that the goal of training Y subnetworks from X is to ensure that all X subnetworks converge as closely as possible. Therefore, reasonable conditions can be set to ensure that the selected P anchor networks are evenly distributed within the first supernetwork. This prevents the anchor networks from having overly similar structures, allowing the P anchor networks to contain subnetworks with more diverse structures. Thus, when the training of the Y subnetworks, including the P anchor networks, converges, all X subnetworks have essentially converged.

[0187] by Figure 8Taking the first supernetwork as an example, if d, e, scale, and the type of header remain unchanged, and only the value of k is varied to select anchor networks, the network structures of the selected anchor networks will be relatively similar. Therefore, to ensure a uniform distribution of anchor networks within the first supernetwork, anchor networks can be selected from multiple dimensions. That is, the values ​​of multiple parameters such as k, d, e, and scale can be varied as much as possible to select P anchor networks.

[0188] Based on the above description, it can be seen that the first method is used to select Y subnetworks. Accordingly, based on preset conditions, P subnetworks are selected from the X subnetworks as anchor networks. The Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks. The first subnetwork is the subnetwork with the largest number of neurons in the X subnetworks, and the second network is the subnetwork with the smallest number of neurons in the X subnetworks. The P anchor networks are the subnetworks that meet the conditions in the X subnetworks, and the Q non-anchor networks are the subnetworks other than the P anchor networks in the X subnetworks. P and Q are both positive integers less than X.

[0189] It should be noted that the embodiment of the present application does not specifically limit the proportion of anchor networks and non-anchor networks in the Y sub-networks. For example, if the value of Y is 8, the Y sub-networks may include 4 anchor networks and 4 non-anchor networks.

[0190] In the embodiment of the present application, since the Y sub-networks include an anchor network selected based on conditions, the parameters of the first super-network are updated according to the training results of the Y sub-networks, so that the parameters of the first super-network can ensure that the anchor network and other anchor networks with similar network structures to the anchor network have better performance.

[0191] based on Figure 8 As can be seen from the relevant description, different subnetworks can be obtained by changing the scaling ratio of the number of channels of the target network. Therefore, the scaling ratio of the number of channels of the target network can be changed to select P anchor networks.

[0192] Exemplarily, the range of variation is related to multiple scaling ratios of the number of channels of the target network; the Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belong to the X subnetworks, and the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, the Q non-anchor networks are other subnetworks among the X subnetworks except the P anchor networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0193] In an embodiment of the present application, the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, so that the P anchor networks include sub-networks of more network structures, thereby enabling training of sub-networks of more network structures, thereby improving the training effect of the first super network.

[0194] Since the Y sub-networks have been described above, the Y sub-networks in the embodiment of the present application can be understood by referring to the relevant description of the Y sub-networks in the above example.

[0195] It should be noted that the multiple preset ratios can be adjusted according to actual needs. For example, the multiple preset ratios may include 0.25, 0.5, 0.75 and 1. Figure 8 Taking block 2 of stage 1 in the hypernetwork shown as an example, the set number Cout of the output channels of the dimensionality reduction convolution is 4. When the scaling ratios of the number of channels of the target network are 0.25, 0.5, 0.75 and 1 respectively, the actual number of output channels of the dimensionality reduction convolution in block 2 can be obtained as 1, 2, 3, and 4 anchor networks respectively. Correspondingly, the scaling ratios of the number of channels of block 2 in these anchor networks can also be said to be 0.25, 0.5, 0.75 and 1 respectively.

[0196] In an embodiment of the present application, by changing the scaling ratio scale of the number of channels of the target network and selecting P anchor networks, P anchor networks with different scaling ratios scale of the number of channels of the target network can be obtained, thereby ensuring the diversity of the P anchor networks.

[0197] In order to make the P anchor networks evenly distributed, as another example, in the P anchor networks, the scaling ratio of the number of channels is such that the number of anchor networks in each scaling ratio in at least two scaling ratios is the same.

[0198] by Figure 8 Taking the supernet shown in FIG5 as an example, the scaling ratios of the number of channels of the target network in the P anchor networks are 0.25, 0.5, 0.75 and 1 respectively, and the number of anchor networks is the same.

[0199] In an embodiment of the present application, in the P anchor networks, the scaling ratio of the number of channels of the target network is the same as the number of anchor networks of each preset ratio, so that the selected P anchor networks are more evenly distributed in the first super network, thereby ensuring the adequacy of the training of the sub-networks of each scaling ratio.

[0200] In this method, training the Y subnetworks typically includes multiple rounds of training. Step 102 includes: after each round of training, the parameters of the first supernetwork are updated based on the training results of the Y subnetworks and using a cumulative gradient method. Specifically, the parameters of the Y subnetworks are first determined based on the initial parameters of the first supernetwork, and then the Y subnetworks are trained in a first round based on the parameters of the Y subnetworks. After the first round of training, the parameters of the first supernetwork are updated based on the training results of the Y subnetworks and using a cumulative gradient method. Then, the Y subnetworks are trained in a second round using the updated parameters of the first supernetwork, and the parameters of the first supernetwork are updated based on the training results of the second round of training of the Y subnetworks. Multiple rounds of training are repeated until the Y subnetworks converge.

[0201] Second method:

[0202] By changing the value of the elastic kernel size k, Y sub-networks are selected from X sub-networks. Figure 8 Taking the first supernetwork shown as an example, for any stage, the elastic depth d is 4; for any block, the elastic width e is 6. The elastic kernel size k is changed to 7, 5, and 3, respectively, to obtain Y subnetworks.

[0203] In this case, P of the Y subnetworks include a first block, where the first block is a block in at least one stage, and P is a positive integer less than or equal to Y. The elastic kernel size k of the depthwise convolution in the first block of the P subnetworks includes multiple preset elastic kernel sizes, where the multiple preset elastic kernel sizes can be set according to actual needs and are not specifically limited in this application. For example, the multiple preset elastic kernel sizes can be 7, 5, and 3, respectively.

[0204] In an embodiment of the present application, by changing the value of the elastic kernel size k, Y subnetworks are selected from X subnetworks, so that the Y subnetworks include subnetworks with multiple elastic kernel sizes, thereby ensuring the uniformity of the distribution of the Y subnetworks in the first supernetwork, so as to realize the training of subnetworks with multiple elastic kernel sizes and improve the training effect.

[0205] It is understandable that in order to ensure the uniformity of the distribution of the Y sub-networks, as a feasible approach, in the P sub-networks, the elastic core size k of the first block is the same as the number of sub-networks with each preset elastic core size.

[0206] After selecting Y subnetworks using this method, the Y subnetworks can be trained sequentially in descending order of k values. Step 102 can then include updating the parameters of the first supernetwork after each subnetwork is trained. For example, the process of training the Y subnetworks includes first training the subnetwork with a 7*7 elastic kernel to convergence, then updating the parameters of the first supernetwork for the first time; then training the subnetwork with a 5*5 elastic kernel to convergence, then updating the parameters of the first supernetwork for the second time; and finally training the subnetwork with a 3*3 elastic kernel to convergence, then updating the parameters of the first supernetwork for the third time.

[0207] It is understood that after selecting Y subnetworks using this method, it is not limited to training the Y subnetworks sequentially in descending order of k values. Other methods can also be used to train the Y subnetworks. For example, a subnetwork with an elastic core size of 7*7, a subnetwork with an elastic core size of 5*5, or a subnetwork with an elastic core size of 3*3 can be randomly selected for training.

[0208] Based on the above description, we know that Y sub-networks share parameters. Figure 9 The case of sharing parameters among Y sub-networks is described by taking the elastic kernel size as an example.

[0209] Figure 9 The convolution kernels of 7*7, 5*5 and 3*3 are shown. The sub-network with the elastic kernel size of 7*7 includes Figure 9 The sub-network with a 7*7 convolution kernel and a 5*5 elastic kernel size includes Figure 9 The sub-network with a 5*5 convolution kernel and a 3*3 elastic kernel size includes Figure 9 After training the subnetwork with a 7*7 elastic kernel, a 7*7 convolution kernel is obtained. When training the subnetwork with a 5*5 elastic kernel, the 5*5 portion inside the 7*7 convolution kernel is used as the 5*5 convolution kernel in the 5*5 subnetwork. Similarly, when training the subnetwork with a 3*3 elastic kernel, the 3*3 portion inside the 5*5 convolution kernel is used as the 3*3 convolution kernel in the 3*3 subnetwork. This shows that the 7*7 subnetwork, the 5*5 subnetwork, and the 3*3 subnetwork share the same convolution kernel.

[0210] Since the shared parameters of Y sub-networks may reduce the performance of some of the Y sub-networks, before training the sub-network with a smaller elastic kernel, the parameters of the convolution kernel of the sub-network with a larger elastic kernel can be slightly adjusted, and the sub-network with a smaller elastic kernel can be trained based on the adjusted convolution kernel parameters.

[0211] For example, before training a sub-network with an elastic kernel size of 5*5, the 5*5 part inside the 7*7 convolution kernel can be multiplied by the preset kernel conversion matrix to obtain a 5*5 convolution kernel of the sub-network with an elastic kernel size of 5*5. The 5*5 convolution kernel obtained after multiplying the kernel conversion matrix is ​​used to train the sub-network with an elastic kernel size of 5*5, thereby reducing the impact of the shared convolution kernel on the performance of Y sub-networks.

[0212] The third method:

[0213] By changing the value of elastic depth d, Y sub-networks are selected from X sub-networks. Figure 8 Taking the first supernetwork shown as an example, for any block, the elastic kernel size k is 7 and the elastic width e is 6; for any stage, the elastic depth d is changed to 4, 3, and 2, respectively, to obtain Y subnetworks.

[0214] In this case, P of the Y subnetworks include a first stage, where the first stage is one of at least one stage, and P is a positive integer less than or equal to Y. The elastic depth d in the first stage of the P subnetworks includes multiple preset elastic depths, where the multiple preset elastic depths can be set based on actual needs and are not specifically limited in this embodiment of the application. For example, the multiple preset elastic depths can be set to 4, 3, and 2, respectively.

[0215] It is understandable that, in order to ensure uniform distribution of the Y sub-networks, as a feasible approach, in the P sub-networks, the elastic depth d of the first stage is the same as the number of sub-networks with each preset elastic depth.

[0216] Similar to the elastic kernel size k, the Y sub-networks can be trained in descending order of the elastic depth d. Accordingly, step 102 includes: after each sub-network is trained, the parameters of the first super-network are updated in sequence.

[0217] For example, the process of training Y sub-networks includes: first training the sub-network with an elastic depth d value of 4, and then updating the parameters of the first super-network for the first time; then training the sub-network with an elastic depth d value of 3, and then updating the parameters of the first super-network for the second time; finally training the sub-network with an elastic depth d value of 2, and then updating the parameters of the first super-network for the third time.

[0218] In an embodiment of the present application, by changing the value of the elastic depth d, Y subnetworks are selected from X subnetworks, so that the Y subnetworks include subnetworks of multiple elastic depths, thereby ensuring the uniformity of the distribution of the Y subnetworks in the first supernetwork, so as to achieve training of subnetworks of multiple elastic depths and improve the training effect.

[0219] Fourth method:

[0220] By changing the value of the elastic width e, Y sub-networks are selected from X sub-networks. Figure 8 Taking the first supernetwork shown as an example, for any stage, the elastic depth d is 4; for any block, the elastic kernel size k is 7, and the elastic width e is changed to 6, 4, and 3 respectively, thus obtaining Y subnetworks.

[0221] In this case, P of the Y subnetworks include a first block, where the first block is a block in at least one stage, and P is a positive integer less than or equal to Y. The elastic width e of the dimensionality-raising point convolution in the first block of the P subnetworks includes multiple preset elastic widths, where the multiple preset elastic widths can be set according to actual needs and are not specifically limited in this application. For example, the values ​​of the multiple preset elastic widths can be 6, 4, and 3, respectively.

[0222] It is understandable that, in order to ensure uniform distribution of the Y sub-networks, as a feasible approach, in the P sub-networks, the elastic width e of the first block is the same as the number of sub-networks with each preset elastic width.

[0223] Similar to the elastic kernel size k, the Y sub-networks can be trained in descending order of the elastic width e. Accordingly, step 102 includes: updating the parameters of the first super-network in sequence after each sub-network is trained.

[0224] For example, the process of training Y sub-networks includes: first training the sub-network with an elastic width e of 6, and then updating the parameters of the first super-network for the first time; then training the sub-network with an elastic width e of 4, and then updating the parameters of the first super-network for the second time; finally training the sub-network with an elastic width e of 3, and then updating the parameters of the first super-network for the third time.

[0225] In the process of training Y sub-networks in descending order of the elastic width e, after training the sub-network with a larger elastic width e, the importance of each channel in the sub-network can be calculated first, and then the channels with higher importance can be selected to form a sub-network with a smaller elastic width e and train it.

[0226] For example, after training the sub-network with an elastic width e of 6, calculate the importance of the 6 channels in the sub-network, and then select the 4 channels with higher importance to form a sub-network with an elastic width e of 4 for training; similarly, the same method can be used to train the sub-network with an elastic width e of 3.

[0227] In an embodiment of the present application, by changing the value of the elastic width e, Y subnetworks are selected from X subnetworks, so that the Y subnetworks include subnetworks with multiple elastic widths, thereby ensuring the uniformity of the distribution of the Y subnetworks in the first supernetwork, so as to achieve training of subnetworks with multiple elastic widths and improve the training effect.

[0228] The fifth method:

[0229] By changing the value of the scaling ratio of the number of channels of the target network, Y sub-networks are selected from X sub-networks. Figure 8 Taking the first super network shown as an example, for any stage, the elastic depth d is 4; for any block, the elastic kernel size k is 7, and the elastic width e is 6; the scaling ratio scale of the number of channels of the target network is changed so that the scaling ratio scale of the number of channels is 0.25, 0.5, 0.75 and 1, respectively, to obtain Y sub-networks.

[0230] Based on the above description, it can be seen that in the embodiment of the present application, the range of variation is related to multiple scaling ratios of the number of channels of the target network; the scaling ratio of the number of channels of the Y sub-networks includes at least two of the multiple scaling ratios.

[0231] It should be noted that multiple scaling ratios can be set according to actual needs, and this application does not impose specific restrictions on this. For example, the values ​​of the multiple scaling ratios can be 0.25, 0.5, 0.75 and 1 respectively.

[0232] In an embodiment of the present application, the scaling ratio of the number of channels of the Y sub-networks includes at least two of the multiple scaling ratios. Compared with the scaling ratio of the number of channels of the Y sub-networks including only one scaling ratio, this implementation method enables the Y sub-networks to include sub-networks with more network structures, thereby improving the training effect of the first super network.

[0233] In one implementation, in the Y sub-networks, the scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same.

[0234] Since the scaling ratio of the number of channels is the same as the number of sub-networks in each scaling ratio in at least two scaling ratios, the adequacy of the training of the sub-networks in each scaling ratio can be guaranteed.

[0235] The sixth method:

[0236] By changing the type of head, Y sub-networks are selected from X sub-networks. For example, Figure 8Taking the first super network shown as an example, a subnetwork with a header of the second version of the mobile terminal network is selected from the X subnetworks, and a subnetwork with a header of the third version of the mobile terminal network is selected from the X subnetworks, thereby obtaining Y subnetworks.

[0237] At this time, the Y sub-networks include a sub-network whose header is the second version of the mobile terminal network and a sub-network whose header is the third version of the mobile terminal network.

[0238] In the embodiment of the present application, the Y sub-networks include a sub-network whose head is the head of the mobile terminal network of the second version, and a sub-network whose head is the head of the mobile terminal network of the third version, so that both the training of the sub-network including the head of the mobile terminal network of the second version and the training of the sub-network including the head of the mobile terminal network of the third version can be achieved.

[0239] In one implementation, among the Y sub-networks, the number of sub-networks including headers of the second version of the mobile terminal network is the same as the number of sub-networks including headers of the third version of the mobile terminal network.

[0240] Since the number of sub-networks including the head of the second version of the mobile terminal network is the same as the number of sub-networks including the head of the third version of the mobile terminal network, the sufficiency of the training of the sub-network including the head of the second version of the mobile terminal network is guaranteed, and the sufficiency of the training of the sub-network including the head of the third version of the mobile terminal network can also be guaranteed.

[0241] In the embodiment of the present application, at least two of the six methods described above may be used to select Y sub-networks from X sub-networks. For example, in the first method described above, the conditions for determining the anchor network may be related to the elastic depth d, the elastic kernel size k, the elastic width e, the scaling ratio of the number of channels scale, and the type of head. In this case, it can be understood that the first method is combined with the other five methods to select Y sub-networks from X sub-networks.

[0242] For another example, the second and third methods can be combined, that is, the elastic depth d and elastic kernel size k are simultaneously changed to obtain Y sub-networks. Specifically, the elastic depth d is 4, 3, or 2, the elastic kernel size k is 7, 5, or 3, and the elastic width e is 6. Based on these conditions, Y sub-networks are selected from X sub-networks.

[0243] The second method, the third method, and the fourth method may also be combined to obtain Y sub-networks. For details, please refer to the relevant description of the combination of the second method and the third method.

[0244] Furthermore, multiple types of training data can be used to train the Y sub-networks, thereby ensuring that the converged Y sub-networks have good performance when processing various types of training data. For example, if the training data is image data, images of different resolutions can be selected as training data.

[0245] In addition, it should be noted that Y sub-networks can be selected from X sub-networks for training multiple times, wherein the Y sub-networks selected each time can be completely the same, partially the same, or completely different; accordingly, step 102 needs to be performed multiple times, that is, step 102 is performed once each time after Y sub-networks are selected from X sub-networks for training.

[0246] Step 103: construct a second supernetwork based on the parameters of the first supernetwork, where the second supernetwork includes M branch networks, each of which contains X subnetworks, and any one of the X subnetworks is contained in only one branch network. All subnetworks in the same branch network share parameters, and M is an integer greater than 1.

[0247] Based on the foregoing description, it can be seen that since the X subnetworks share parameters, when the parameter value increases, the performance of some subnetworks will improve, while the performance of other subnetworks will deteriorate; when the parameter value decreases, similarly, the performance of some subnetworks will improve, while the performance of other subnetworks will deteriorate. Therefore, when updating the parameters in the first supernetwork, it is impossible to take into account the performance of all X subnetworks, which may result in poor performance of some of the X subnetworks. To this end, after executing step 102, the X subnetworks can be divided into M groups, and a second supernetwork including M branch networks can be constructed, each branch network containing a group of subnetworks, that is, each branch network contains at least one of the X subnetworks; any subnetwork in the X subnetworks is only contained in one branch network, that is, different branch networks do not contain the same subnetwork.

[0248] In this way, the network structures of the sub-networks in the same branch network may overlap, so all sub-networks in the same branch network share parameters, while sub-networks in different branch networks do not share parameters, thereby reducing the impact of shared parameters on the performance of X sub-networks.

[0249] It should be noted that the X subnetworks can be divided into M groups by various methods, which are not specifically limited in the embodiments of the present application. For example, the X subnetworks can be divided into M groups by dividing the search space into M parts. Specifically, the search space can be divided into M parts by dividing the elastic depth d, elastic kernel size k, elastic width e, the scaling ratio scale of the number of channels of the target network, and the value range of the head type in the search space.

[0250] The following combination Figures 10 to 16 The relationship between the first supernetwork, the X subnetworks, and the second supernetwork will be described.

[0251] Specifically, if Figure 10 As shown, Figure 10 The neural network in is the first super network. Figure 10 The first super network shown can be obtained Figure 11 、 Figure 12 as well as Figure 13 The three sub-networks are divided into three groups, and correspondingly, a second super-network containing three branch networks is constructed. Figure 14 、 Figure 15 and Figure 16 The second super network is shown respectively, and Figure 14 Shown Figure 11 The position of the subnetwork in the second supernetwork, Figure 15 Shown Figure 12 The position of the subnetwork in the second supernetwork, Figure 16 Shown Figure 13 The position of the subnetwork in the second supernetwork.

[0252] Step 104 : Based on the training results of Z sub-networks among the X sub-networks, update the parameters of the branch networks to which the Z sub-networks belong, where Z is a positive integer less than or equal to X.

[0253] Before executing step 104, Z sub-networks may be selected from the X sub-networks for training. It should be noted that the method for selecting Z sub-networks from the X sub-networks for training is the same as the method for selecting Y sub-networks from the X sub-networks for training. For details, please refer to the relevant description of step 102.

[0254] Taking the first method in step 102 as an example, after constructing the first supernetwork, before updating the parameters of the first supernetwork based on the training results of Y subnetworks among the X subnetworks, P subnetworks that meet the conditions are selected from the X subnetworks as P anchor networks, and the other subnetworks among the X subnetworks except the P anchor networks are Q non-anchor networks. Before training, Z subnetworks can be selected from the X subnetworks for training by the following method: all anchor networks among the P anchor networks are selected, at least one non-anchor network is randomly selected from the non-anchor networks, and then the subnetwork with the largest number of neurons and the subnetwork with the smallest number of neurons are selected from the X subnetworks, thereby obtaining Z subnetworks.

[0255] Accordingly, the Z subnetworks include at least one of the first subnetwork, the second subnetwork, P anchor point networks, and Q non-anchor point networks. The first subnetwork is the subnetwork with the largest number of neurons in the X subnetworks, the second network is the subnetwork with the smallest number of neurons in the X subnetworks, the P anchor point networks are the subnetworks that meet the conditions in the X subnetworks, and the Q non-anchor point networks are the other subnetworks in the X subnetworks except the P anchor point networks. P and Q are both positive integers less than X.

[0256] Since the Z sub-networks include an anchor network selected based on preset conditions, the parameters of the second super-network are updated according to the training results of the Z sub-networks, so that the parameters of the second super-network can ensure that the anchor network and other anchor networks with similar network structures to the anchor network have better performance.

[0257] If P anchor networks are selected by changing a scaling ratio scale of the number of channels of the target network, then correspondingly, a range of variation is related to multiple scaling ratios of the number of channels of the target network; the Z subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belong to the X subnetworks, and the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, the Q non-anchor networks are other subnetworks among the X subnetworks excluding the P anchor networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0258] Because the scaling ratio of the number of channels in the P anchor networks includes at least two of the multiple scaling ratios, the P anchor networks include more sub-networks of the network structure, thereby enabling training of more sub-networks of the network structure, thereby improving the training effect of the first super-network. In one implementation, the scaling ratio of the number of channels in the P anchor networks is such that the number of anchor networks in each of the at least two scaling ratios is the same.

[0259] In the P anchor networks, the scaling ratio of the number of channels of the target network is the same as the number of anchor networks of at least two preset ratios, thereby ensuring the adequacy of training of the sub-networks of each scaling ratio.

[0260] Other descriptions about selecting Z sub-networks from X sub-networks for training using the first method are not repeated here in this embodiment of the present application.

[0261] Taking the fifth method in step 102 as an example, the scaling ratio of the number of channels of the Z sub-networks includes at least two of the multiple scaling ratios.

[0262] Since the scaling ratio of the number of channels of the Z sub-networks includes at least two of the multiple scaling ratios, compared to the case where the scaling ratio of the number of channels of the Z sub-networks includes only one scaling ratio, this implementation method enables the Z sub-networks to include sub-networks with more network structures, thereby enabling training of sub-networks with more network structures, thereby improving the training effect of the first super network.

[0263] In one implementation, in the Z sub-networks, the scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same.

[0264] The scaling ratio of the number of channels is such that the number of sub-networks in each scaling ratio in at least two scaling ratios is the same, thereby ensuring the adequacy of the training of the sub-networks in each scaling ratio.

[0265] Taking the sixth method in step 102 as an example, the selected Z sub-networks include the sub-network with the header of the second version of the mobile terminal network and the sub-network with the header of the third version of the mobile terminal network.

[0266] Since the Z sub-networks include a sub-network whose head is the head of the mobile terminal network of the second version and a sub-network whose head is the head of the mobile terminal network of the third version, it is possible to train the sub-network including the head of the mobile terminal network of the second version and the sub-network including the head of the mobile terminal network of the third version.

[0267] In one implementation, among the Y sub-networks, the number of sub-networks including headers of the second version of the mobile terminal network is the same as the number of sub-networks including headers of the third version of the mobile terminal network.

[0268] Since the number of sub-networks including the head of the second version of the mobile terminal network is the same as the number of sub-networks including the head of the third version of the mobile terminal network, it is possible to ensure the sufficiency of the training of the sub-network including the head of the second version of the mobile terminal network and the sufficiency of the training of the sub-network including the head of the third version of the mobile terminal network.

[0269] It should be noted that since sub-networks in the same branch network share parameters, while sub-networks in different branch networks do not share parameters; therefore, unlike step 102, in the process of executing step 104, the parameters of the second super network need to be updated according to the branch networks to which the Z sub-networks belong.

[0270] For example, the second super network includes a first branch network, a second branch network, and a third branch network, and 10 sub-networks are selected from these three branch networks for training, wherein the first branch network, the second branch network, and the third branch network include 5, 3, and 2 sub-networks, respectively; then step 104 includes: updating the parameters of the first branch network according to the 5 sub-networks, updating the parameters of the second branch network according to the 3 sub-networks, and updating the parameters of the third branch network according to the 2 sub-networks.

[0271] Similarly, since the Z sub-networks include an anchor network selected based on conditions, the parameters of the second super-network are updated according to the training results of the Z sub-networks, so that the parameters of the second super-network can ensure that the anchor network and other anchor networks with similar network structures have better performance.

[0272] Based on the relevant descriptions of steps 101 to 104, the process of training the neural network in the embodiment of the present application can be summarized as follows: first, the first super network with shared parameters is trained, and then the second super network including the branch network is trained with multiple branches. The process of training the first super network is the same as that of training the second super network. Figure 17 First, a first supernetwork with shared parameters is constructed. Training data is then obtained, and some subnetworks are selected from the X subnetworks. These subnetworks are trained, and the supernetwork weights are updated based on the training results. These steps are repeated. Next, a second supernetwork with multiple branches is constructed, and the training steps are repeated, ultimately resulting in a converged second supernetwork.

[0273] In an embodiment of the present application, a first supernetwork comprising X subnetworks is first constructed, wherein the X subnetworks share parameters. Then, Y subnetworks of the X subnetworks are trained, and the parameters of the first supernetwork are updated based on the training results of the Y subnetworks, thereby ensuring that the X subnetworks have good performance. Thereafter, a second supernetwork comprising M branch networks is constructed, wherein each branch network comprises at least one of the X subnetworks. Then, Z subnetworks of the X subnetworks are trained, and the parameters of the branch networks to which the Z subnetworks belong are updated based on the training results of the Z subnetworks. Because all subnetworks in the same branch network share parameters, and based on the training results of one subnetwork of the Z subnetworks, only the parameters of the branch network to which the subnetwork belongs are updated, without causing changes in the parameters of subnetworks in other branch networks, thereby reducing mutual interference between the parameters of the X subnetworks, further improving the performance of the X subnetworks, and thus achieving better performance of the neural network ultimately deployed on the terminal device.

[0274] For ease of understanding, the method for training a neural network provided in an embodiment of the present application is introduced below using a specific example.

[0275] This example includes:

[0276] The first step is to construct a first super network. For details, please refer to the relevant description of step 101. The first super network is constructed based on the search space, which is constructed based on the target network, and the first super network includes the target network.

[0277] In the second step, images of different resolutions are used as samples for training the first hypernetwork.

[0278] The third step is to train the target network until convergence.

[0279] The fourth step is to adjust the parameters of the first super network from the dimension of the elastic kernel size k, that is, select sub-networks with different elastic kernel sizes from the first super network and train them until convergence. For details, please refer to the second method in the relevant description of step 102.

[0280] The fifth step is to adjust the parameters of the first super network from the dimension of elastic depth d, that is, select sub-networks of different elastic depths from the first super network and train them until convergence. For details, please refer to the third method in the relevant description of step 102.

[0281] The sixth step is to adjust the parameters of the first super network from the dimension of the elastic width e, that is, select sub-networks with different elastic widths from the first super network and train them until convergence. For details, please refer to the fourth method in the relevant description of step 102 for understanding.

[0282] In the seventh step, the parameters of the first super network are adjusted using the improved sandwich strategy, that is, a sub-network including at least one anchor network and at least one non-anchor network is selected from the first super network and trained until convergence. For details, please refer to the third method in the relevant description of step 101.

[0283] In the eighth step, the parameters of the first super network are adjusted in a multi-branch manner. Specifically, a second super network including multiple branch networks is constructed based on the first super network, and then a sub-network is selected from the second super network and trained until convergence. For details, please refer to the relevant instructions of step 103 for understanding.

[0284] In the ninth step, the parameters of the second super network are adjusted from the dimension of the head. For details, please refer to the sixth method in the relevant description of step 102.

[0285] In order to highlight the effect of the method for training a neural network provided by the embodiment of the present application, the method for training a neural network provided by the present application and the existing Once For All (OFA) algorithm are used for training respectively, and the search space is expanded by about 8 times, covering 10 20The training results of the sub-network are shown in the following table:

[0286]

[0287] As can be seen from the above table, the method for training a neural network provided in the present application is used for training. Compared with the existing training once for multiple deployments (Once For All, OFA) algorithm for training, the lower limit of the parameter range of the sub-network is reduced by 57%, the lower limit of the FLOPs range of the sub-network is reduced by 83%, and the lower limit of the accuracy range of the sub-network is also reduced by 20%. Therefore, the sub-network trained by the method provided in the embodiment of the present application covers an ultra-lightweight neural network.

[0288] See also Figure 18 , Figure 18 Schematic diagram of an embodiment of a device for constructing an index in an embodiment of the present application. Figure 18 As shown, an embodiment of the present application provides a device for training a neural network, comprising:

[0289] A construction unit 201 is configured to construct a first supernetwork, wherein the first supernetwork includes X subnetworks, and the X subnetworks share parameters, where X is an integer greater than 1;

[0290] A training unit 202 is configured to update parameters of the first supernetwork based on training results of Y subnetworks out of the X subnetworks, where Y is a positive integer less than or equal to X;

[0291] A construction unit 201 is configured to construct a second supernetwork based on the parameters of the first supernetwork, where the second supernetwork includes M branch networks, each of the M branch networks includes X subnetworks, and any one of the X subnetworks is included in only one branch network. All subnetworks in the same branch network share parameters, and M is an integer greater than 1.

[0292] The training unit 202 is configured to update parameters of the branch networks to which the Z subnetworks belong based on the training results of the Z subnetworks among the X subnetworks, where Z is a positive integer less than or equal to X.

[0293] Among them, the function of the construction unit 201 can be understood by referring to the relevant description of step 101 in the above text, and the function of the training unit 202 can be understood by referring to the relevant description of step 102 in the above text.

[0294] In one implementation,

[0295] The Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks. The first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks are the subnetworks that meet the conditions among the X subnetworks, and the Q non-anchor networks are the subnetworks other than the P anchor networks among the X subnetworks. P and Q are both positive integers less than X.

[0296] In one implementation,

[0297] The Z subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks. The first subnetwork is the subnetwork with the largest number of neurons in the X subnetworks, the second network is the subnetwork with the smallest number of neurons in the X subnetworks, the P anchor networks are the subnetworks that meet the conditions in the X subnetworks, and the Q non-anchor networks are the other subnetworks in the X subnetworks except the P anchor networks. P and Q are both positive integers less than X.

[0298] In one implementation, the training unit 202 is further configured to select P sub-networks that meet the conditions from the X sub-networks as P anchor networks.

[0299] In one implementation, the construction unit 201 is configured to establish a search space based on a target network, where the search space indicates a range of changes in a network structure of the target network; and construct a first super network based on the search space.

[0300] In one implementation,

[0301] The target network includes an input layer, a head, and at least one stage located before the input layer and the head, each stage includes at least one block, and each block includes a dimension-increasing point convolution, a depth-wise convolution, and a dimension-reducing convolution; the range of variation is related to at least one of the following parameters: a scaling ratio of the number of channels of the target network and a type of head; the scaling ratio of the number of channels of the target network indicates at least one of the following ratios: a ratio of the actual number of output channels of the input layer to a set number of output channels, a ratio of the actual number of output channels of the head to a set number of output channels, and a ratio of the actual number of output channels of the dimension-reducing convolution in each block to a set number of output channels of the dimension-reducing convolution.

[0302] In an implementation, the target network is a second version of the mobile terminal network, and the type of the header is a header of the second version of the mobile terminal network or a header of a third version of the mobile terminal network; the second version of the mobile terminal network includes five stages, and each stage of the second version of the mobile terminal network sequentially includes two blocks, three blocks, four blocks, three blocks, and three blocks from a direction close to the input layer to a direction away from the input layer; the third version of the mobile terminal network includes five stages, and each stage of the third version of the mobile terminal network sequentially includes two blocks, three blocks, four blocks, two blocks, and three blocks from a direction close to the input layer to a direction away from the input layer.

[0303] In an implementation,

[0304] In an implementation, the variation range is related to a plurality of scaling ratios of the number of channels of the target network; the scaling ratio of the number of channels of the Y sub-networks includes at least two of the plurality of scaling ratios and / or the scaling ratio of the number of channels of the Z sub-networks includes at least two of the plurality of scaling ratios.

[0305] In an implementation,

[0306] The Y sub-networks include a sub-network with a header of the second version of the mobile terminal network and a sub-network with a header of the third version of the mobile terminal network; and / or the Z sub-networks include a sub-network with a header of the second version of the mobile terminal network and a sub-network with a header of the third version of the mobile terminal network.

[0307] In an implementation,

[0308] The variation range is related to a plurality of scaling ratios of the number of channels of the target network; the Y sub-networks include at least one of a first sub-network, a second sub-network, P anchor point networks, and Q non-anchor point networks, the first sub-network is a sub-network with the largest number of neurons in the X sub-networks, the second network is a sub-network with the smallest number of neurons in the X sub-networks, the P anchor point networks belong to the X sub-networks, and the scaling ratio of the number of channels of the P anchor point networks includes at least two of the plurality of scaling ratios, the Q non-anchor point networks are other sub-networks in the X sub-networks except the P anchor point networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0309] In one implementation, the range of variation is related to multiple scaling ratios of the number of channels of the target network; the Z subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork is the subnetwork with the largest number of neurons among the X subnetworks, the second network is the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belong to the X subnetworks, and the scaling ratio of the number of channels of the P anchor networks includes at least two of the multiple scaling ratios, the Q non-anchor networks are other subnetworks among the X subnetworks excluding the P anchor networks, P is a positive integer less than X and greater than 1, and Q is a positive integer less than X.

[0310] See also Figure 19 , Figure 19 This is a schematic diagram of the structure of the server in the embodiment of the present application.

[0311] like Figure 19 As shown, an embodiment of the present application also provides an embodiment of a server. The server may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 2022 (for example, one or more processors) and memory 2032, and one or more storage media 2030 (for example, one or more mass storage devices) storing application programs 2042 or data 2044. Among them, the memory 2032 and the storage medium 2030 can be short-term storage or persistent storage. The program stored in the storage medium 2030 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the training device. Furthermore, the central processing unit 2022 can be configured to communicate with the storage medium 2030 to execute a series of instruction operations in the storage medium 2030 on the server 2000.

[0312] The server 2000 may also include one or more power supplies 2026, one or more wired or wireless network interfaces 2050, one or more input and output interfaces 2058, and / or one or more operating systems 2041, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0313] In the embodiment of the present application, the central processing unit 2022 is used to execute the above Figure 7 The steps of the method in the illustrated embodiment will not be described in detail here.

[0314] Exemplarily, the CPU 2022 may perform the following steps:

[0315] Constructing a first super network, wherein the first super network includes X sub-networks, and the X sub-networks share parameters, where X is an integer greater than 1;

[0316] Update the parameters of the first supernetwork based on the training results of Y subnetworks among the X subnetworks, where Y is a positive integer less than or equal to X;

[0317] Constructing a second supernetwork based on the parameters of the first supernetwork, where the second supernetwork includes M branch networks, each of the M branch networks includes X subnetworks, and any subnetwork of the X subnetworks is contained in only one branch network. All subnetworks in the same branch network share parameters, and M is an integer greater than 1.

[0318] Based on the training results of Z sub-networks among the X sub-networks, update the parameters of the branch networks to which the Z sub-networks belong, where Z is a positive integer less than or equal to X.

[0319] In the embodiment of the present application, the specific functional module division in the central processing unit 222 can be the same as the aforementioned Figure 18 The division of functional modules such as construction units and training units described in is similar and will not be repeated here.

[0320] The embodiment of the present application also provides a chip or chip system, which includes at least one processor and a communication interface, the communication interface and the at least one processor are interconnected through a line, and the at least one processor is used to run a computer program or instruction to execute the aforementioned Figure 7 The steps of the method in the illustrated embodiment will not be described in detail here.

[0321] The communication interface in the chip may be an input / output interface, a pin or a circuit, etc.

[0322] The present application also provides a first embodiment of a chip or chip system. The chip or chip system described above in the present application further includes at least one memory, wherein the at least one memory stores instructions. The memory may be a storage unit within the chip, such as a register, a cache, etc., or a storage unit of the chip (e.g., a read-only memory, a random access memory, etc.).

[0323] An embodiment of the present application further provides a computer storage medium for storing computer software instructions used by the above-mentioned control device, including a program designed for executing a server.

[0324] The server can be as mentioned above Figure 18 The described apparatus for training a neural network.

[0325] The present application also provides a computer program product, which includes computer software instructions that can be loaded by a processor to implement the aforementioned Figure 7 The process in the method shown.

[0326] See also Figure 20 , Figure 20 This is a structural diagram of the model deployment system in an embodiment of the present application.

[0327] The embodiment of the present application also provides a model deployment system, including: a terminal device 400 and a server 500;

[0328] The server 500 is used to execute the method for training a neural network in the embodiment shown in FIG. 7 , and to select a subnet from the second super network and deploy it in the terminal device 400 .

[0329] Specifically, the server 500 is used to:

[0330] Constructing a first super network, wherein the first super network includes X sub-networks, and the X sub-networks share parameters, where X is an integer greater than 1;

[0331] Update the parameters of the first supernetwork based on the training results of Y subnetworks among the X subnetworks, where Y is a positive integer less than or equal to X;

[0332] Constructing a second supernetwork based on the parameters of the first supernetwork, where the second supernetwork includes M branch networks, each of the M branch networks includes X subnetworks, and any subnetwork of the X subnetworks is contained in only one branch network. All subnetworks in the same branch network share parameters, and M is an integer greater than 1.

[0333] Based on the training results of Z sub-networks among the X sub-networks, update the parameters of the branch networks to which the Z sub-networks belong, where Z is a positive integer less than or equal to X.

[0334] Then, the server 500 is further configured to select a subnet from the second super network and deploy it in the terminal device 400 .

[0335] Specifically, the server 500 randomly samples N sub-networks from the second super-network and obtains performance indicators of the N sub-networks, where the performance indicators may include the accuracy, inference latency, and number of parameters of the sub-networks, and N is a positive integer.

[0336] Based on the above explanation, we can see that the number of parameters is only related to the subnetwork itself, while the inference latency is also related to the operating environment, specifically the terminal device on which it is deployed. Therefore, we can obtain the inference latency of each layer of neurons on the target device and then construct an inference latency lookup table. Since multiple layers of neurons have a serial dependency relationship, we can calculate the inference latency of the neural network on the target device by adding the inference latency layer by layer.

[0337] The accuracy of a sub-network is determined by its network structure, parameters, and validation dataset. Therefore, the parameters of the corresponding sub-network can be obtained from the second super-network, and then inference and calculation can be performed on the validation dataset to obtain the accuracy of the sub-network.

[0338] Finally, the server 500 can construct a performance lookup table based on the performance indicators of the N sub-networks. The primary key value of the performance lookup table can be the unique code of the sub-network, and the attribute value is the accuracy, inference delay and number of parameters of the corresponding sub-network.

[0339] As an implementable approach, the unique code can be constructed using the structural parameters of the subnetwork. Specifically, 3557357335773355-4634466633464334-42424-0.25-v2 is used as the unique code of a subnetwork, where the unique code represents that the elastic depths d of the five stages in the subnetwork are 4, 2, 4, 2, and 4, respectively; the elastic kernel sizes k of the 16 blocks in the five stages are 3, 5, 5, 7, 3, 5, 7, 3, 3, 5, 7, 7, 3, 3, 5, 5, respectively; the elastic widths e of the 16 blocks in the five stages are 4, 6, 3, 4, 4, 6, 6, 6, 3, 3, 4, 6, 4, 3, 3, and 4, respectively; the scaling ratio of the number of channels in the subnetwork is 0.25; and the header type is the header of the second version of the mobile network.

[0340] See Figure 21 , Figure 21 A schematic diagram of the structure of a terminal device 400 provided in an embodiment of the present application.

[0341] like Figure 21As shown, the terminal device 400 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0342] It should be understood that the structure illustrated in the embodiments of the present application does not constitute a specific limitation on the terminal device 400. In other embodiments of the present application, the terminal device 400 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0343] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0344] The controller may be the nerve center and command center of the terminal device 400. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0345] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0346] In some embodiments, the processor 110 may include one or more interfaces. The interfaces may include an inter-integrated circuit (IC) interface, an inter-integrated circuit sound (IS) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface.

[0347] It is understood that the interface connection relationship between the modules illustrated in the embodiment of the present application is merely an illustrative illustration and does not constitute a structural limitation on the terminal device 400. In other embodiments of the present application, the terminal device 400 may also adopt a different interface connection method from the above embodiment, or a combination of multiple interface connection methods.

[0348] The charging management module 140 is configured to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from the wired charger via the USB interface 130.

[0349] The power management module 141 is used to connect the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140 to provide power to the processor 110, the internal memory 121, the external memory, the display 194, the camera 193, and the wireless communication module 160.

[0350] The wireless communication function of the terminal device 400 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0351] In some feasible implementations, the terminal device 400 can use wireless communication functions to communicate with other devices. For example, the terminal device 400 can communicate with a second electronic device, establish a screen projection connection with the second electronic device, and output screen projection data to the second electronic device. The screen projection data output by the terminal device 400 can be audio or video data.

[0352] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 400 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0353] The mobile communication module 150 can provide solutions for wireless communications including 1G / 3G / 4G / 5G applied to the terminal device 400. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 2. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0354] The modem processor may include a modulator and a demodulator. The modulator is used to modulate the low-frequency baseband signal to be transmitted into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After being processed by the baseband processor, the low-frequency baseband signal is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to the speaker 170A, the receiver 170B, etc.) or displays an image or video through the display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0355] The wireless communication module 160 can provide a solution for wireless communication including wireless local area networks (WLAN) (e.g., wireless fidelity (Wi-Fi) network), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR) technology, etc. applied to the terminal device 400. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 1, frequency-modulates and filters the electromagnetic wave signals, and transmits the processed signals to the processor 110. The wireless communication module 160 can also receive signals to be transmitted from the processor 110, frequency-modulate them, amplify them, and radiate them as electromagnetic waves via the antenna 2.

[0356] In some embodiments, antenna 1 of terminal device 400 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, so that terminal device 400 can communicate with a network and other devices via wireless communication technology. The wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology. The GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0357] Terminal device 400 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0358] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, terminal device 400 may include one or N display screens 194, where N is a positive integer greater than 1.

[0359] In some feasible implementations, the display screen 194 may be used to display various interfaces output by the system of the terminal device 400. For details on various interfaces output by the terminal device 400, reference may be made to the relevant descriptions of subsequent embodiments.

[0360] The terminal device 400 can realize the shooting function through the ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0361] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and converted into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0362] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 400 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0363] Digital signal processors are used to process digital signals. In addition to processing digital image signals, they can also process other digital signals.

[0364] Video codecs are used to compress or decompress digital video. Terminal device 400 may support one or more video codecs. This allows terminal device 400 to play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG1, MPEG3, and MPEG4.

[0365] The NPU is a neural network (NN) computing processor. Drawing on the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it rapidly processes input information and can continuously self-learn. The NPU enables intelligent cognitive applications in the terminal device 400, such as image recognition, face recognition, speech recognition, and text comprehension.

[0366] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 400. The external memory card communicates with the processor 110 via the external memory interface 120 to implement data storage functions. For example, files such as music and videos can be stored on the external memory card.

[0367] The internal memory 121 can be used to store computer executable program codes, which include instructions. The processor 110 executes various functional applications and data processing of the terminal device 400 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the terminal device 400 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0368] The terminal device 400 can implement audio functions such as music playback and recording through the audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor. In some feasible embodiments, the audio module 170 can be used to play the sound corresponding to the video. For example, when the display screen 194 displays the video playback screen, the audio module 170 outputs the sound of the video playback.

[0369] The audio module 170 is used to convert digital audio information into analog audio signals for output, and is also used to convert analog audio input into digital audio signals.

[0370] The speaker 170A, also called a "horn", is used to convert audio electrical signals into sound signals.

[0371] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals.

[0372] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0373] The headphone jack 170D is used to connect a wired headphone and can be the USB interface 130 or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0374] The pressure sensor 180A is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor 180A can be provided on the display screen 194. The gyroscope sensor 180B can be used to determine the motion posture of the terminal device 400. The air pressure sensor 180C is used to measure air pressure.

[0375] Accelerometer 180E can detect the magnitude of acceleration of terminal device 400 in all directions (including three or six axes). When terminal device 400 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify terminal posture, enabling applications such as switching between landscape and portrait modes and pedometers.

[0376] The distance sensor 180F is used to measure distance.

[0377] The ambient light sensor 180L is used to sense the brightness of the ambient light.

[0378] The fingerprint sensor 180H is used to collect fingerprints.

[0379] The temperature sensor 180J is used to detect temperature.

[0380] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be provided on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be provided on the surface of the terminal device 400, in a location different from that of the display screen 194.

[0381] The buttons 190 include a power button, a volume button, etc. The buttons 190 may be mechanical buttons or touch buttons. The terminal device 400 may receive key inputs and generate key signal inputs related to user settings and function control of the terminal device 400.

[0382] Motor 191 can generate vibration prompts.

[0383] The indicator 192 may be an indicator light, which may be used to indicate the charging status, power level changes, messages, missed calls, notifications, etc.

[0384] The SIM card interface 195 is used to connect a SIM card.

[0385] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0386] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0387] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0388] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0389] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A method for training a neural network, characterized in that include: Constructing a first supernetwork, wherein the first supernetwork includes X subnetworks, the X subnetworks share parameters, and X is an integer greater than 1; Updating parameters of the first supernetwork based on training results of Y subnetworks among the X subnetworks, where Y is a positive integer less than or equal to X, and the training results of the Y subnetworks are obtained by training the Y subnetworks using data, the data including at least one of the following: images, speech, and text; Constructing a second supernetwork based on the parameters of the first supernetwork, where the second supernetwork includes M branch networks, the M branch networks include the X subnetworks, any one of the X subnetworks is included in only one branch network, all subnetworks in the same branch network share parameters, and M is an integer greater than 1; Based on the training results of Z sub-networks among the X sub-networks, the parameters of the branch networks to which the Z sub-networks belong are updated, where Z is a positive integer less than or equal to X.

2. The method according to claim 1, characterized in that The Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor point networks, and Q non-anchor point networks, the first subnetwork being the subnetwork with the largest number of neurons among the X subnetworks, the second subnetwork being the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor point networks being those that meet the conditions among the X subnetworks, the Q non-anchor point networks being other subnetworks among the X subnetworks excluding the P anchor point networks, and P and Q being positive integers less than X.

3. The method according to claim 1 or 2, characterized in that The Z subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor point networks, and Q non-anchor point networks, the first subnetwork being the subnetwork with the largest number of neurons in the X subnetworks, the second subnetwork being the subnetwork with the smallest number of neurons in the X subnetworks, the P anchor point networks being the subnetworks that meet the conditions in the X subnetworks, the Q non-anchor point networks being the other subnetworks in the X subnetworks except the P anchor point networks, and P and Q being positive integers less than X.

4. The method according to claim 2, characterized in that After constructing the first supernetwork and before updating the parameters of the first supernetwork based on the training results of Y subnetworks among the X subnetworks, the method further includes: P sub-networks that meet the conditions are selected from the X sub-networks as the P anchor networks.

5. The method according to any one of claims 1 to 2, characterized in that The constructing of the first super network comprises: Establishing a search space based on a target network, wherein the search space indicates a range of changes in a network structure of the target network; A first supernetwork is constructed based on the search space.

6. The method according to claim 5, characterized in that The target network includes an input layer, a head, and at least one stage located before the input layer and the head, each stage includes at least one block, and each block includes a dimension-increasing point convolution, a depth-wise convolution, and a dimension-reducing convolution; The variation range is related to at least one of the following parameters: a scaling ratio of the number of channels of the target network and a type of head; The scaling ratio of the number of channels of the target network indicates at least one of the following ratios: the ratio of the actual number of output channels of the input layer to the set number of output channels, the ratio of the actual number of output channels of the head to the set number of output channels, and the ratio of the actual number of output channels of the dimensionality reduction convolution in each block to the set number of output channels of the dimensionality reduction convolution.

7. The method according to claim 6, characterized in that The target network is a second-version mobile network or a third-version mobile network; When the target network is a second version mobile network, the type of the header is a header of the second version mobile network or a header of the third version mobile network; When the target network is a third-version mobile network, the type of the header is a header of the second-version mobile network or a header of the third-version mobile network; The second version of the mobile network includes five stages, and each stage in the second version of the mobile network includes two blocks, three blocks, four blocks, three blocks, and three blocks in a direction from close to the input layer to far away from the input layer; The third version of the mobile network includes five stages, and each stage in the third version of the mobile network includes two blocks, three blocks, four blocks, two blocks, and three blocks in sequence from close to the input layer to away from the input layer.

8. The method according to claim 6, characterized in that The variation range is related to a plurality of scaling ratios of the number of channels of the target network; The scaling ratio of the number of channels of the Y sub-networks includes at least two of the plurality of scaling ratios and / or the scaling ratio of the number of channels of the Z sub-networks includes at least two of the plurality of scaling ratios.

9. The method according to claim 7, characterized in that The Y sub-networks include a sub-network whose header is the header of the second version of the mobile terminal network, and a sub-network whose header is the header of the third version of the mobile terminal network; and / or The Z sub-networks include a sub-network whose head is the head of the second version of the mobile terminal network and a sub-network whose head is the head of the third version of the mobile terminal network.

10. The method according to claim 6, characterized in that The variation range is related to a plurality of scaling ratios of the number of channels of the target network; The Y subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork being the subnetwork with the largest number of neurons among the X subnetworks, the second subnetwork being the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belonging to the X subnetworks, and a scaling ratio of the number of channels of the P anchor networks including at least two of the multiple scaling ratios, the Q non-anchor networks being other subnetworks among the X subnetworks excluding the P anchor networks, P being a positive integer less than X and greater than 1, and Q being a positive integer less than X.

11. The method according to claim 6, characterized in that The variation range is related to a plurality of scaling ratios of the number of channels of the target network; The Z subnetworks include at least one of a first subnetwork, a second subnetwork, P anchor networks, and Q non-anchor networks, the first subnetwork being the subnetwork with the largest number of neurons among the X subnetworks, the second subnetwork being the subnetwork with the smallest number of neurons among the X subnetworks, the P anchor networks belonging to the X subnetworks, and a scaling ratio of the number of channels of the P anchor networks including at least two of the multiple scaling ratios, the Q non-anchor networks being other subnetworks among the X subnetworks excluding the P anchor networks, P being a positive integer less than X and greater than 1, and Q being a positive integer less than X.

12. A device for training a neural network, characterized in that: include: A construction unit, configured to construct a first supernetwork, wherein the first supernetwork includes X subnetworks, the X subnetworks share parameters, and X is an integer greater than 1; a training unit, configured to update parameters of the first supernetwork based on training results of Y subnetworks among the X subnetworks, where Y is a positive integer less than or equal to X, and the training results of the Y subnetworks are obtained by training the Y subnetworks using data, the data comprising at least one of the following: images, speech, and text; a construction unit, configured to construct a second supernetwork based on the parameters of the first supernetwork, wherein the second supernetwork includes M branch networks, the M branch networks include the X subnetworks, and any subnetwork of the X subnetworks is included in only one branch network, all subnetworks in the same branch network share parameters, and M is an integer greater than 1; A training unit is configured to update parameters of branch networks to which the Z subnetworks belong based on training results of the Z subnetworks among the X subnetworks, where Z is a positive integer less than or equal to X.

13. A server, characterized in that: include: At least one processor and a memory, the memory storing computer-executable instructions that can be run on the processor, and when the computer-executable instructions are executed by the processor, the server executes the method as described in any one of claims 1 to 11 above.

14. A computer-readable storage medium storing one or more computer-executable instructions, characterized in that: When the computer-executable instructions are executed by a processor, the processor performs the method according to any one of claims 1 to 11.

15. A model deployment system, characterized in that: include: Terminal devices and servers; The server is used to execute the method according to any one of claims 1 to 11, and is used to select a subnet from the second super network to be deployed in the terminal device.

Images