Method and device for training super network

Abstract

The embodiment of the invention provides a method and a device for training a super network. The method comprises the following steps: acquiring a to-be-trained super network comprising a plurality of sub-networks; training the to-be-trained super network by sequentially executing multiple iterative operations, wherein the current iterative operation comprises the following steps: sampling the super network to obtain a sub-network sequence; and training the sub-network sequence based on the super-network, and updating the super-network based on a loss function of the sub-network sequence and a weight attenuation factor corresponding to a current iterative operation, the weight attenuation factor decreasing as an iterative order of the current iterative operation increases. The weight attenuation factor is dynamically adjusted based on the iteration order, it can be guaranteed that the super network is fully trained, and a small quantization error can be achieved.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence technology, and in particular to a method and device for training a hypernetwork. Background Art

[0002] Neural Architecture Search (NAS) is an important part of automatic network structure design. A good NAS method can search for high-precision network structures that meet resource requirements in a short time. In NAS: First, a super network containing all possible sub-networks is constructed and trained. Then, a suitable sub-network is searched from the super network to serve as a neural network model for performing actual deep learning tasks.

[0003] In order to achieve faster inference speed and smaller model during deployment, the floating-point precision model searched by NAS is usually quantized to fixed point (for example, quantized to INT8). In the process of converting from floating point to fixed point, quantization error is often introduced due to information loss, resulting in precision loss. Summary of the invention

[0004] The embodiments of the present invention provide a method and device for training a hypernetwork.

[0005] The technical solution of the embodiment of the present invention is as follows:

[0006] A method for training a hypernetwork, comprising:

[0007] Obtain a super network to be trained that contains multiple sub-networks;

[0008] The super network to be trained is trained by sequentially performing multiple iteration operations, wherein the current iteration operation includes:

[0009] Sampling the super network to obtain a sub-network sequence;

[0010] Based on the super network, the sub-network sequence is trained, wherein the sub-network sequence is trained based on the loss function of the sub-network sequence and the weight decay factor corresponding to the current iterative operation.

[0011] The super network is updated, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0012] In one embodiment, it includes:

[0013] In response to determining that the current super network meets a preset convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value, stopping the execution of the iterative operation;

[0014] The current hypernetwork is determined to be a trained hypernetwork.

[0015] In one embodiment, it includes:

[0016] Based on a predetermined first search condition, performing a sub-network search on the trained super-network, wherein the first search condition does not include a fixed-point quantization strategy;

[0017] Performing model quantization processing on the searched sub-network that meets the first search condition;

[0018] Deploy the sub-network after the model quantization processing.

[0019] In one embodiment, it includes:

[0020] Based on a predetermined second search condition, performing a sub-network search on the trained super network, wherein the second search condition includes a fixed-point quantization strategy;

[0021] The searched sub-networks meeting the second search condition are deployed.

[0022] In one embodiment, the fixed-point quantization strategy includes at least one of the following:

[0023] The number of quantization bits;

[0024] Quantification range calibration strategy.

[0025] In one embodiment, it includes:

[0026] Characterizing the weight decay factor with a function expression including a predetermined maximum value of the weight decay factor, a predetermined minimum value of the weight decay factor, an iteration order of a current iteration operation, and a total number of iterations;

[0027] The function expression includes at least one of the following:

[0028] Trigonometric function expression; linear function expression; logarithmic function expression; exponential function expression.

[0029] A device for training a hypernetwork, comprising:

[0030] An acquisition module, used to acquire a super network to be trained including multiple sub-networks;

[0031] The training module is used to train the super network to be trained by sequentially performing multiple iteration operations, wherein the current iteration operation includes:

[0032] Sampling the super network to obtain a sub-network sequence;

[0033] Based on the super network, the sub-network sequence is trained, wherein based on the sub-network

[0034] The loss function of the network sequence and the weight decay factor corresponding to the current iteration operation are used to

[0035] The super network is updated, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0036] In one embodiment, the training module is used to stop executing the iterative operation in response to determining that the current supernetwork meets a preset convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value; and determine that the current supernetwork is a trained supernetwork.

[0037] In one embodiment, it includes:

[0038] A deployment module is used to search for sub-networks on the trained super network based on a predetermined first search condition, wherein the first search condition does not include a fixed-point quantization strategy; perform model quantization processing on the sub-networks that meet the search condition; and deploy the quantized sub-networks.

[0039] In one embodiment, it includes:

[0040] A deployment module is used to search for a sub-network on the trained super-network based on a predetermined second search condition, wherein the second search condition includes a fixed-point quantization strategy; and deploy the searched sub-network that meets the search condition.

[0041] It can be seen from the above technical solution that in the embodiment of the present invention, a super network to be trained including multiple sub-networks is obtained; the super network to be trained is trained by sequentially performing multiple iteration operations, wherein the current iteration operation includes: sampling the super network to obtain a sub-network sequence; based on the super network, the sub-network sequence is trained, wherein the super network is updated based on the loss function of the sub-network sequence and the weight decay factor corresponding to the current iteration operation, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases. Therefore, dynamically adjusting the weight decay factor based on the iteration order can not only ensure that the super network is fully trained, but also achieve a smaller quantization error.

[0042] In addition, by adding a quantization strategy when searching for subnetworks, subnetworks that are friendly to quantization deployment can be directly searched out. Moreover, the training process of the supernetwork is decoupled from the search process of the subnetwork, the supernetwork is fully optimized, and the searched subnetwork has good quantization accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 is an exemplary flow chart of a method for training a hypernetwork according to an embodiment of the present invention.

[0044] Figure 2 It is a schematic diagram of a first exemplary process of deploying a sub-network according to an embodiment of the present invention.

[0045] Figure 3 It is a schematic diagram of a second exemplary process of deploying a sub-network according to an embodiment of the present invention.

[0046] Figure 4 It is a schematic diagram of an exemplary process of performing an image classification task according to an embodiment of the present invention.

[0047] Figure 5 It is an exemplary structural diagram of an apparatus for training a hypernetwork according to an embodiment of the present invention.

[0048] Figure 6 is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the objectives, technical solutions and advantages of the present invention more clear, the present invention is further described in detail below with reference to the accompanying drawings.

[0050] For the sake of brevity and intuitiveness in description, the scheme of the present invention is explained below by describing several representative implementations. A large number of details in the implementations are only used to help understand the scheme of the present invention. However, it is obvious that the technical scheme of the present invention may not be limited to these details when implemented. In order to avoid unnecessarily obscuring the scheme of the present invention, some implementations are not described in detail, but only a framework is given. Hereinafter, "including" means "including but not limited to", and "according to..." means "at least according to..., but not limited to only according to...". Due to the language habits of Chinese, when the number of a component is not specifically specified below, it means that the component can be one or more, or can be understood as at least one.

[0051] The applicant found that: in the prior art, although the floating point accuracy of the sub-network searched by NAS meets the requirements, the accuracy after quantization may not meet expectations.

[0052] The applicant analyzed the causes of the above technical problems and found that: in the prior art, in order to fully train the hypernetwork, the weight decay factor in the training process is usually set to a low value (for example, set to 10 -5 ). Moreover, the weight decay factor is fixed to this lower value in all iterations of the hypernetwork. A lower weight decay factor makes the quantization range of the weight larger, so the weight update is more affected by the gradient, which introduces a larger quantization error. However, if the weight decay factor is increased to reduce the quantization error, the hypernetwork may not be fully trained.

[0053] It can be seen that the use of a relatively low and fixed weight decay factor in the prior art makes it difficult to achieve a good compromise between sufficient training of the super network and a relatively small quantization error. In particular: the weights of the sub-networks in the super network are often coupled, and the weights of the smaller sub-networks are usually part of the weights of the larger sub-networks. Therefore, when a relatively low and fixed weight decay factor is used, the weights of the common parts of the two coupled sub-networks will obtain a larger gradient, while the weights of the non-common parts will obtain a smaller gradient, which makes the quantization range differences of different channels of the sub-networks larger, and the accuracy drop is more obvious.

[0054] The applicant further discovered that: during the entire training phase of the hypernetwork, the weight decay factor actually has a significant impact on the second norm of the weight (which can characterize the quantization range of the weight) only in the early stage of training, and the impact on the second norm of the weight gradually decreases in the later stage of training. Based on this discovery, the embodiments of the present invention overcome the common thinking of generally adopting a weight decay factor with a low value and a fixed value, and dynamically adjust the weight decay factor in time sequence, wherein a larger weight decay factor is used in the early stage of training (corresponding to a smaller iteration order) to reduce the quantization range of the weight, and a smaller weight decay factor is used in the later stage of training (corresponding to a larger iteration order) to reduce the quantization error and improve the model accuracy, which can ensure that the hypernetwork is fully trained and achieve a smaller quantization error, thereby achieving a good compromise between the full training of the hypernetwork and the smaller quantization error.

[0055] The above disclosure details the technical defects existing in the prior art, the causes of the technical defects and the thinking and analysis process of overcoming the technical defects. In fact, the cognition of the above technical defects is not common knowledge in the field, but a novel discovery made by the applicant in research. In addition, the cause tracing of the technical defects and the thinking and analysis process of overcoming the technical defects are also the gradual analysis results of the applicant in the actual research process, and are not common knowledge in the field.

[0056] Figure 1 FIG. 1 is an exemplary flow chart of a method for training a hypernetwork according to an embodiment of the present invention. Figure 1 As shown, the method includes:

[0057] Step 101: Obtain a super network to be trained that includes multiple sub-networks.

[0058] The supernetwork to be trained may be pre-built. The supernetwork to be trained contains all possible subnetworks. Each layer in the supernetwork may contain multiple network structure units in the network structure search space. Here, the network structure unit may be formed by a single network layer (e.g., a single convolutional layer, a single cyclic unit in a recurrent neural network), or may be formed by a combination of multiple network layers (e.g., a convolutional block formed by connecting a convolutional layer, a batch normalization layer, a nonlinear layer, etc.). In a supernetwork, each network structure unit may be connected to all network structure units of the previous and next layers. The parameters of the supernetwork to be trained may be randomly initialized, or the parameters of the supernetwork to be trained may be initialized to set values. Optionally, the supernetwork to be trained may be a supernetwork pre-trained based on sample data.

[0059] Step 102: The super network to be trained is trained by sequentially executing multiple iteration operations, wherein the current iteration operation includes: sampling the super network to obtain a sub-network sequence; based on the super network, training the sub-network sequence, wherein the super network is updated based on the loss function of the sub-network sequence and a weight decay factor corresponding to the current iteration operation, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0060] The role of the weight decay factor is to suppress overfitting of the model, thereby improving the generalization of the model. For example, the weight decay factor adds a penalty of the L2 norm of the model weight to the loss function so that the model weight is not too large, thereby reducing the complexity of the model and suppressing overfitting of the model. During the training process of the hypernetwork, the parameters in the hypernetwork (including weight parameters, bias parameters, convolution kernels, etc.) are optimized through multiple iterations.

[0061] Each iteration operation includes the following sub-steps:

[0062] Sub-step 1: Sample the super network to obtain a sub-network sequence.

[0063] In sub-step 1, several sub-networks can be sampled from the super-network to form a sub-network sequence. Specifically, a random sampling method can be used to extract multiple network layers from the super-network, extract the connection relationship between these network layers, and obtain the parameters of the sampled several sub-networks from the super-network.

[0064] Sub-step 2: Based on the super network, the sub-network sequence is trained, wherein the super network is updated based on the loss function of the sub-network sequence and the weight decay factor corresponding to the current iterative operation, wherein the weight decay factor is set to decrease as the iteration order of the current iterative operation increases.

[0065] Specifically: In each iteration, forward propagation and back propagation processes are performed on each sub-network in the sub-network sequence to obtain the accumulated gradient, and then the parameters of the super network are updated based on the accumulated gradient.

[0066] For example, the loss function L of the hypernetwork in each iteration operation can be determined as follows:

[0067]

[0068] Where: A′ is any subnetwork in the subnetwork sequence; A is the subnetwork search space in the supernetwork; D trn is the sub-network training data; N is the super-network; W A′ is the weight of A′; L trn (A′,W A′ ; N, D trn ) is the loss function of the sub-network; γ is the weight decay factor; W is the weight of the super-network; R(W) is the second normal form of W.

[0069] It can be seen that the loss function based on the sub-network sequence (L trn (A′,W A ,;N,D trn )) and the weight decay factor (γ) corresponding to the current iterative operation can determine the loss function L of the hypernetwork, so the hypernetwork can be updated based on the loss function L of the hypernetwork.

[0070] The above lists a typical example of the loss function L of the hypernetwork in each iteration operation. Similarly, the objective function W of the hypernetwork can be listed as * Typical examples include:

[0071] W * =min W (E A′∈A [L trn (A′,W A′ ; N, D trn )]+γR(W))

[0072] Where: A is the subnetwork search space in the supernetwork; W is the weight of the supernetwork; γ is the weight decay factor; R(W) is the second normal form of W; A′ is any subnetwork in the subnetwork sequence; D trn is the sub-network training data; N is the super-network; W A′ is the weight of A′; L trn (A′,W A′ ; N, D trn ) is the loss function of the sub-network; W * is the optimized supernet weight; E is the expectation.

[0073] It should be noted that: unlike the prior art in which γ adopts a fixed smaller value, the γ in the embodiment of the present invention is a dynamic value and is set to decrease as the iteration order of the current iteration operation increases. Specifically, in the first round of iteration (correspondingly, the iteration order is 1), the value of γ is the largest; the value of γ in the second round of iteration (correspondingly, the iteration order is 2)) is smaller than the value of γ in the first round of iteration; the value of γ in the third round of iteration (correspondingly, the iteration order is 3)) is smaller than the value of γ in the second round of iteration. By analogy, in the last round of iteration, the value of γ is smaller than the value of γ in all previous iterations.

[0074] In one embodiment, the method includes: characterizing the weight decay factor with a function expression including a predetermined maximum value of the weight decay factor, a predetermined minimum value of the weight decay factor, an iteration order of the current iteration operation, and a total number of iterations. The function expression includes at least one of the following: a trigonometric function expression; a linear function expression; a logarithmic function expression; an exponential function expression, etc.

[0075] For example, assuming that the maximum value of the predetermined weight decay factor is γ max , the minimum value of the predetermined weight attenuation factor is γ min , the iteration order of the current iteration operation is t, and the total number of iterations is T; the weight attenuation factor γ of the current iteration order can be determined t ,in:

[0076]

[0077] The value range of t is [1, T]. Therefore, in the first iteration (t is equal to 1), γ1 = γ max ; At the Tth iteration (t equals T), γ T =γ min ; and between the first round of iteration and the Tth round of iteration, the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0078] The above uses the trigonometric function expression as an example to exemplarily describe the expression of the weight attenuation factor. Those skilled in the art will appreciate that this description is merely exemplary and is not intended to limit the scope of protection of the embodiments of the present invention.

[0079] Sub-step 3: When the convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value, if yes, the iteration is stopped, otherwise the next iteration is executed.

[0080] In one embodiment, the method further includes: in response to determining that the current hypernetwork satisfies a preset convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value, stopping the iterative operation; and determining that the current hypernetwork is a trained hypernetwork. For example, the convergence condition may include but is not limited to at least one of the following: the parameters of the hypernetwork have been updated in the most recent iterations and the update rate is lower than a preset update rate threshold; the accuracy of the hypernetwork reaches a preset accuracy threshold, etc.

[0081] It can be seen that during the training process of the hypernetwork, dynamically adjusting the weight attenuation factor can effectively control the quantization range of the weight while improving the floating-point accuracy of the hypernetwork, thereby reducing the drop points after quantization. Assuming that an image detection task is to be performed, inputting the image into the trained hypernetwork will improve the accuracy of image detection. Those skilled in the art will understand that this is only used for illustration and should not be construed as a limitation of the present invention. For example, target recognition, classification, detection, etc. can all use the trained hypernetwork of the present invention for task processing, and can improve the accuracy of processing.

[0082] In one embodiment, the method includes: performing a sub-network search on a trained super network based on a predetermined first search condition, wherein the first search condition does not include a fixed-point quantization strategy; performing model quantization processing on the searched sub-network that meets the first search condition; and deploying the sub-network processed by the model quantization processing.

[0083] In this implementation, for the first search condition that does not include a fixed-point quantization strategy, a subnetwork that meets the first search condition is directly output from the trained supernetwork, and then the subnetwork that meets the first search condition is subjected to model quantization. In this implementation, the search condition is relatively simple, so a subnetwork that meets the search condition can be easily searched out, and the searched subnetwork also has good quantization accuracy.

[0084] In one embodiment, the method includes: performing a sub-network search on a trained super network based on a predetermined second search condition, wherein the second search condition includes a fixed-point quantization strategy; and deploying the searched sub-network that meets the second search condition.

[0085] In one embodiment, the fixed-point quantization strategy includes at least one of the following: the number of quantization bits; a quantization range calibration strategy, and the like.

[0086] Example:

[0087]

[0088] Where: MQ is the quantitative strategy; ∈ is the constraint (such as the size of the sub-network, the time the sub-network runs, etc.); Accuracy is the accuracy function; D valis the training data; A* is the optimal subnetwork that meets the conditions; A′ is any subnetwork in the subnetwork sequence; A is the subnetwork search space in the supernetwork; W A′ is the weight of A′; W is the weight of the hypernetwork; N is the hypernetwork; Cons(N(A′, W A′ )) is the pair of A′ and W A′ constraints.

[0089] In this implementation, since the search conditions already include a fixed-point quantization strategy, the searched subnetwork does not need to perform model quantization processing and can be directly deployed. The searched subnetwork also has good quantization accuracy. Therefore, the subnetwork can be quickly deployed according to the needs of the deployed device. When searching the network structure, the influence of model quantization is taken into account, and the searched subnetwork is a quantization deployment-friendly model.

[0090] In summary, in the embodiments of the present invention, when training a super network, the floating point accuracy of the super network can be improved based on the dynamically adjusted weight attenuation factor, and the quantization range of the weight can be effectively controlled, thereby reducing the point drop after quantization. Moreover, the training process of the super network is decoupled from the search process of the sub network, the super network is fully optimized, and the searched sub network has good quantization accuracy. In addition, by adding a quantization strategy when searching for a sub network, a sub network that is friendly to quantization deployment can be directly searched out.

[0091] Figure 2 FIG. 1 is a schematic diagram of a first exemplary process of deploying a sub-network according to an embodiment of the present invention. Figure 2 As shown, the method includes:

[0092] Step 201: Obtain a super network to be trained that includes multiple sub-networks.

[0093] Step 202: The super network to be trained is trained by sequentially executing multiple iteration operations, wherein the current iteration operation includes: sampling the super network to obtain a sub-network sequence; based on the super network, training the sub-network sequence, wherein the super network is updated based on a loss function of the sub-network sequence and a weight decay factor corresponding to the current iteration operation, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0094] Step 203: Based on a predetermined first search condition, a sub-network search is performed on the trained super network, wherein the first search condition does not include a quantization strategy.

[0095] Step 204: Perform model quantization processing on the searched sub-network that meets the first search condition. For example, linear quantization or nonlinear quantization can be used to perform model quantization processing on the sub-network.

[0096] Step 205: Deploy the sub-network after the model quantization processing.

[0097] Figure 3 FIG. 1 is a schematic diagram of a second exemplary process of deploying a sub-network according to an embodiment of the present invention. Figure 3 As shown, the method includes:

[0098] Step 301: Obtain a super network to be trained that includes multiple sub-networks.

[0099] Step 302: The super network to be trained is trained by sequentially executing multiple iteration operations, wherein the current iteration operation includes: sampling the super network to obtain a sub-network sequence; based on the super network, the sub-network sequence is trained, wherein the super network is updated based on the loss function of the sub-network sequence and the weight decay factor corresponding to the current iteration operation, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0100] Step 303: Based on a predetermined second search condition, a sub-network search is performed on the trained super network, wherein the second search condition includes a quantization strategy.

[0101] Step 304: deploy the searched sub-network that meets the second search condition.

[0102] The technical solution for training a hypernetwork proposed in the embodiment of the present invention can be applied to a variety of specific application scenarios, such as image detection scenarios, image classification scenarios, image recognition scenarios, and the like.

[0103] The following uses an image classification scenario as an example to illustrate an implementation of the present invention. Figure 4 FIG. 1 is a schematic diagram of an exemplary process of performing an image classification task according to an embodiment of the present invention. Figure 4 As shown, the method includes:

[0104] Step 401: Obtain a super network to be trained that includes multiple sub-networks adapted to perform image classification tasks.

[0105] Here, based on prior knowledge, multiple sub-networks suitable for performing image classification tasks can be determined, such as Resnet network, Mobilenet network, Inception network, and Densenet, and a super network to be trained containing these sub-networks can be constructed.

[0106] Step 402: The super network to be trained is trained by sequentially executing multiple iteration operations, wherein the current iteration operation includes: sampling the super network to obtain a sub-network sequence; inputting the training data in the image classification task into the sub-network sequence in the super network to train the sub-network sequence, wherein the super network is updated based on the loss function of the sub-network sequence and the weight decay factor corresponding to the current iteration operation, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0107] For further details of step 402, please refer to Figure 1 The above detailed description. Wherein: The training data in the image classification task can have classification labels or not.

[0108] Step 403: Based on the search conditions including the quantization strategy in the image classification task, a sub-network search is performed on the trained super-network.

[0109] For example, the quantization strategy in the image classification task may include: the number of quantization bits in the image classification task and the quantization range calibration strategy in the image classification task, etc.

[0110] The quantization range calibration strategy mentioned in the embodiment of the present invention, for example, minimum and maximum calibration, MSE calibration or cross entropy calibration. The above list is a candidate quantization range calibration strategy, which is only used for illustration and should not be understood as a limitation of the present invention. According to the accuracy results searched out, determine which calibration strategy is used for which specific subnet or task. For example, a subnet with high search accuracy and a combination of quantization strategies are selected.

[0111] Step 404: determine the sub-network that meets the search criteria as an image classification model, and deploy the image classification model.

[0112] Step 405: Perform an image classification task based on the image classification model. For example, input the image to be tested into the image classification model so that the image classification model can classify the image to be tested.

[0113] In the above exemplary process of performing image classification tasks, when training the super network, the dynamically adjusted weight attenuation factor can improve the floating-point accuracy of the super network and effectively control the quantization range of the weights. The image classification model searched from the super network has the advantage of high quantization accuracy, which is conducive to performing high-precision image classification tasks.

[0114] In an optional implementation, the search condition in step 403 may not include the quantization strategy in the image classification task, and the image classification model searched out still has good quantization accuracy. To ensure better image classification accuracy, for the image classification model searched out by the search condition that does not include the quantization strategy, it is preferred to perform quantization model processing before model deployment.

[0115] Figure 5 is an exemplary structural diagram of a device for training a hypernetwork according to an embodiment of the present invention. Figure 5 As shown, the apparatus 500 for training a hypernetwork includes:

[0116] An acquisition module 501 is used to acquire a super network to be trained including multiple sub-networks;

[0117] The training module 502 is used to train the super network to be trained by performing multiple iteration operations in sequence, wherein the current iteration operation includes: sampling the super network to obtain a sub-network sequence; based on the super network, training the sub-network sequence, wherein the super network is updated based on the loss function of the sub-network sequence and the weight decay factor corresponding to the current iteration operation, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

[0118] In one embodiment, the training module 502 is used to stop executing the iterative operation in response to determining that the current hypernetwork meets a preset convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value; and determine that the current hypernetwork is a trained hypernetwork.

[0119] In one embodiment, the device 500 includes a deployment module 503, which is used to search for sub-networks on a trained super network based on a predetermined first search condition, wherein the first search condition does not include a fixed-point quantization strategy; perform model quantization processing on the sub-networks searched out that meet the first search condition; and deploy the quantized sub-networks.

[0120] In this embodiment of the present invention, if the fixed-point quantization strategy is not included, the search speed will be faster. For example, in the case of insufficient iterative computing power, the demand for hardware resources can be reduced and a faster response speed can be guaranteed.

[0121] In one embodiment, the apparatus 500 includes a deployment module 503 for performing a sub-network search on a trained super-network based on a predetermined second search condition, wherein the second search condition includes a fixed-point quantization strategy; and deploying a sub-network found to meet the second search condition.

[0122] In this embodiment of the present invention, if a fixed-point quantization strategy is included, the accuracy of the quantization will be higher. For some tasks that require higher quantization accuracy, such as face recognition unlocking, accurate identification of the target person is required to ensure the safety of family members or property in the unlocking scene. At this time, by adding a fixed-point quantization strategy, the subnetwork searched after the fixed-point quantization can improve the accuracy of task execution, thereby improving security.

[0123] Figure 6 It is an exemplary structural diagram of an electronic device according to an embodiment of the present invention. The electronic device 600 includes: a processor 601 and a memory 602. The processor 601 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 601 may be implemented in at least one hardware form of digital signal processing (DSP), field programmable gate array (FPGA), and programmable logic array (PLA). The processor 601 may also include a main processor and a coprocessor, the main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 601 may be integrated with a graphics processing unit (GPU), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 601 may also include an AI processor, which is used to process computing operations related to machine learning. For example, the AI ​​processor may be implemented as a neural network processor.

[0124] The memory 602 may include one or more computer-readable storage media, which may be non-transitory. The memory 602 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices.

[0125] In some embodiments, the non-transitory computer-readable storage medium in the memory 602 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 601 to implement the method for training a hypernetwork provided in each embodiment of the present disclosure. In some embodiments, the electronic device 600 may also optionally include: a peripheral device interface 603 and at least one peripheral device. The processor 601, the memory 602 and the peripheral device interface 603 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 603 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 604, a touch display screen 605, a camera assembly 606, an audio circuit 607, a positioning assembly 608 and a power supply 609. The peripheral device interface 603 may be used to connect at least one peripheral device related to input / output (I / O) to the processor 601 and the memory 602. In some embodiments, the processor 601, the memory 602, and the peripheral device interface 603 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 601, the memory 602, and the peripheral device interface 603 can be implemented on a separate chip or circuit board, which is not limited in this embodiment.

[0126] The radio frequency circuit 604 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The radio frequency circuit 604 communicates with the communication network and other communication devices through electromagnetic signals. The radio frequency circuit 604 converts the electrical signal into an electromagnetic signal for transmission, or converts the received electromagnetic signal into an electrical signal. Optionally, the radio frequency circuit 604 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 604 can communicate with other terminals through at least one wireless communication protocol. The wireless communication protocol includes but is not limited to: a metropolitan area network, various generations of mobile communication networks (2G, 3G, 4G and 5G), a wireless local area network and / or a wireless fidelity (Wireless Fidelity, Wi-Fi) network. In some embodiments, the radio frequency circuit 804 may also include circuits related to near field communication (NFC), which is not limited in the present disclosure.

[0127] The display screen 605 is used to display the user interface (UI). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 605 is a touch display screen, the display screen 605 also has the ability to collect touch signals on the surface or above the surface of the display screen 605. The touch signal can be input to the processor 601 as a control signal for processing. At this time, the display screen 605 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards. In some embodiments, the display screen 605 can be one, set on the front panel of the electronic device 600; in other embodiments, the display screen 605 can be at least two, respectively set on different surfaces of the electronic device 600 or in a folding design; in some embodiments, the display screen 605 can be a flexible display screen, set on the curved surface or folding surface of the electronic device 600. Even, the display screen 605 can also be set to a non-rectangular irregular shape, that is, a special-shaped screen. The display screen 605 can be made of materials such as liquid crystal display (LCD), organic light-emitting diode (OLED), etc.

[0128] The camera assembly 606 is used to capture images or videos. Optionally, the camera assembly 606 includes a front camera and a rear camera. Typically, the front camera is disposed on the front panel of the terminal, and the rear camera is disposed on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize the panoramic shooting and the virtual reality (VR) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 606 may also include a flash. The flash may be a monochrome temperature flash or a dual color temperature flash. A dual color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0129] The audio circuit 607 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them into the processor 601 for processing, or input them into the radio frequency circuit 604 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the electronic device 600. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert the electrical signal from the processor 601 or the radio frequency circuit 604 into sound waves. The speaker may be a traditional film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert the electrical signal into a sound wave audible to humans, but also convert the electrical signal into a sound wave inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 607 may also include a headphone jack. The positioning component 608 is used to locate the current geographical location of the electronic device 800 to achieve navigation or location-based services (LBS). Positioning component 608 may be a positioning component based on the Global Positioning System (GPS) of the United States, the Beidou system of China, the Grenas system of Russia, or the Galileo system of the European Union. Power supply 609 is used to power various components in electronic device 600. Power supply 609 may be alternating current, direct current, a disposable battery, or a rechargeable battery. When power supply 609 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging.

[0130] Those skilled in the art will appreciate that the above structure does not constitute a limitation on the electronic device 600, and may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements. It should be noted that not all steps and modules in the above processes and structure diagrams are necessary, and certain steps or modules may be ignored according to actual needs. The execution order of each step is not fixed and may be adjusted as needed. The division of each module is only for the convenience of describing the functional division adopted. In actual implementation, a module can be implemented by multiple modules, and the functions of multiple modules can also be implemented by the same module. These modules can be located in the same device or in different devices. The hardware modules in each embodiment can be implemented mechanically or electronically. For example, a hardware module can include a specially designed permanent circuit or logic device (such as a dedicated processor, such as FPGA or ASIC) for completing a specific operation. The hardware module may also include a programmable logic device or circuit (such as a general-purpose processor or other programmable processor) temporarily configured by software for performing a specific operation. As for the specific mechanical method, or the use of a dedicated permanent circuit, or the use of a temporarily configured circuit (such as configured by software) to implement the hardware module, it can be determined based on cost and time considerations.

[0131] The present invention also provides a machine-readable storage medium, storing instructions for making a machine execute such as the method of the present application. Specifically, a system or device equipped with a storage medium can be provided, on which a software program code for realizing the function of any of the above-mentioned embodiments is stored, and the computer (or CPU or MPU) of the system or device reads out and executes the program code stored in the storage medium. In addition, the operating system etc. operated on the computer can also be completed part or all of the actual operation by instructions based on the program code. The program code read out from the storage medium can also be written to the memory provided in the expansion board inserted into the computer or to the memory provided in the expansion unit connected to the computer, and then the CPU etc. installed on the expansion board or the expansion unit are made to perform part and all of the actual operation based on the instructions of the program code, so as to realize the function of any of the above-mentioned embodiments. The storage medium implementation for providing the program code includes a floppy disk, a hard disk, a magneto-optical disk, an optical disk (such as CD-ROM, CD-R, CD-RW, DVD-ROM, DVD-RAM, DVD-RW, DVD+RW), a magnetic tape, a non-volatile memory card and a ROM. Alternatively, the program code can be downloaded from a server computer or a cloud by a communication network.

[0132] The above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for training a hypernetwork, characterized in that: include: Obtain a super network to be trained that contains multiple sub-networks; The super network to be trained is trained by sequentially performing multiple iteration operations, wherein the current iteration operation includes: Sampling the super network to obtain a sub-network sequence; Based on the super network, the sub-network sequence is trained, wherein based on the sub-network The hypernetwork is updated based on the loss function of the hypernetwork sequence and the weight decay factor corresponding to the current iterative operation, wherein the weight decay factor decreases as the iteration order of the current iterative operation increases.

2. The method according to claim 1, characterized in that include: In response to determining that the current super network meets a preset convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value, stopping the execution of the iterative operation; The current hypernetwork is determined to be a trained hypernetwork.

3. The method according to claim 2, characterized in that include: Based on a predetermined first search condition, performing a sub-network search on the trained super-network, wherein the first search condition does not include a fixed-point quantization strategy; Performing model quantization processing on the searched sub-network that meets the first search condition; Deploy the sub-network after the model quantization processing.

4. The method according to claim 2, characterized in that: include: Based on a predetermined second search condition, performing a sub-network search on the trained super network, wherein the second search condition includes a fixed-point quantization strategy; The searched sub-networks meeting the second search condition are deployed.

5. The method according to claim 4, characterized in that The fixed-point quantization strategy includes at least one of the following: The number of quantization bits; Quantification range calibration strategy.

6. The method according to any one of claims 1 to 5, characterized in that include: Characterizing the weight decay factor with a function expression including a predetermined maximum value of the weight decay factor, a predetermined minimum value of the weight decay factor, an iteration order of a current iteration operation, and a total number of iterations; The function expression includes at least one of the following: Trigonometric function expression; linear function expression; logarithmic function expression; exponential function expression.

7. A device for training a hypernetwork, characterized in that: include: An acquisition module, used to acquire a super network to be trained including multiple sub-networks; The training module is used to train the super network to be trained by sequentially performing multiple iteration operations, wherein the current iteration operation includes: Sampling the super network to obtain a sub-network sequence; Based on the super network, the sub-network sequence is trained, wherein based on the sub-network The loss function of the network sequence and the weight decay factor corresponding to the current iteration operation are used to The super network is updated, wherein the weight decay factor decreases as the iteration order of the current iteration operation increases.

8. The device according to claim 7, characterized in that The training module is used to stop executing the iterative operation in response to determining that the current super network meets the preset convergence condition or the total number of iterations of the iterative operation reaches a predetermined set value; and determine that the current super network is a trained super network.

9. The device according to claim 8, characterized in that include: A deployment module, configured to search for a sub-network on the trained super-network based on a predetermined first search condition, wherein the first search condition does not include a fixed-point quantization strategy; Performing model quantization processing on the searched sub-network that meets the first search condition; The quantized sub-network is deployed.

10. The device according to claim 9, characterized in that include: A deployment module, configured to search for a sub-network on the trained super-network based on a predetermined second search condition, wherein the second search condition includes a fixed-point quantization strategy; The searched sub-networks meeting the second search condition are deployed.

Images