A neural network distillation method, a target detection method and device

Abstract

The application discloses a neural network distillation method, a target detection method and device in the field of artificial intelligence, which are used for efficiently obtaining a lightweight model meeting requirements in combination with structure search, pruning or distillation and the like, and have strong generalization ability, and are used for performing target detection tasks or classification tasks and the like. The method comprises the following steps: firstly, a teacher model and a student model are acquired; the student model is evolved to obtain an evolved student model, wherein the evolution comprises: transforming the structure of the student model according to an evolution action to obtain the evolved student model, the evolution action comprises at least one of adding a layer, layer rearrangement or pruning, the adding of the layer indicates that at least one network layer is added in the student model, the layer rearrangement indicates that at least one network layer in the student model is rearranged, and the pruning indicates that the structure of the student model is reduced; and the evolved student model is distilled by using the teacher model to obtain an updated student model.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a neural network distillation method, a target detection method and device. BACKGROUND

[0002] In the field of artificial intelligence, computer vision technology has a wide range of applications in automatic driving, mobile terminals and security monitoring scenarios. The traditional solution to high-level computer vision tasks is based on manually set convolutional neural networks. For example, in target detection, the fastest region-based convolutional neural network (Faster-RCNN) is the most widely used target detection architecture. Faster-RCNN uses a residual network (ResNet) as the backbone network module, a feature pyramid network (FPN) as the multi-level feature extraction module, and finally inputs the features extracted by the front module into the head network (Head) module related to the detection task to obtain the prediction result. The depth (number of layers), width (number of channels) and connection relationship of the ResNet, FPN and Head networks are designed according to human past experience and specific task requirements. Therefore, different neural networks may need to be designed manually in different scenarios, and the generalization ability is weak. SUMMARY

[0003] The present application provides a neural network distillation method, a target detection method and device, which are used to efficiently obtain a model that meets the requirements by combining structure search, pruning or distillation, etc., and have strong generalization ability.

[0004] Therefore, in a first aspect, the present application provides a neural network distillation method, comprising: first obtaining an initial teacher model and a first student model, and the output accuracy of the teacher model is higher than that of the first student model; then evolving the first student model to obtain an evolved first student model, wherein the evolution process can include transforming the structure of the first student model according to an evolution action to obtain the evolved first student model, the evolution action can include at least one of adding a layer, layer rearrangement or pruning, the adding a layer indicates adding at least one network layer in the first student model, the layer rearrangement indicates rearranging at least one network layer in the first student model, and the pruning indicates reducing the structure of the first student model; distilling the evolved first student model using the teacher model to obtain an updated first student model; if the updated first student model meets a preset condition, the updated first student model is used as a target model, and the target model is used to perform a preset task.

[0005] Therefore, in the embodiments of the present application, structural search, pruning and distillation are combined to quickly deform the teacher model or the student model, improve the efficiency of changing the model, and more efficiently find a better lightweight student model. In addition, the trained teacher model is used for knowledge distillation of the student model to quickly complete the training of the student model, thereby improving the efficiency of obtaining the final output model. Moreover, the output accuracy of the student model can be improved by improving the output accuracy of the teacher model. In a possible implementation, if the updated first student model does not meet the preset condition, a second student model is obtained, the second student model is evolved, the teacher model or a new teacher model is used for distillation of the evolved second student model, and a second student model updated again is obtained. In various scenarios such as image segmentation, target detection or classification, the method provided in the present application can be used, that is, a large number of models with different structures are obtained through structural search or pruning, and then the output accuracy of the model can be improved through distillation, so that the optimal student model can be screened, that is, the generalization ability is strong and can adapt to various scenarios.

[0006] Therefore, in the embodiments of the present application, if a student model meeting the preset condition is not obtained, the steps of evolution and distillation can be repeatedly performed until a student model meeting the preset condition is obtained.

[0007] In a possible implementation, the above-mentioned obtaining of the teacher model and the first student model can include: obtaining the teacher model and the first student model according to a model pair list, the model pair list including at least one model pair, each model pair including a structure of a teacher model and a structure of a first student model.

[0008] In the embodiments of the present application, the initial teacher model and student model can be selected by setting the model pair list. Therefore, a model pair with better performance can be selected from the model pair list as the initial teacher model pair to improve the convergence speed of the student model and improve the efficiency of obtaining a student model meeting the preset condition.

[0009] In a possible implementation, the above-mentioned method can further include: if the updated first student model does not meet the preset condition, adding the teacher model and the updated first student model as a model pair to the model pair list.

[0010] In the embodiments of the present application, if the first student model obtained through distillation does not meet the preset condition, the structure of the teacher model and the first student model after distillation is added to the model pair list, so that the model pair included in the model pair list can be referred to when the initial teacher model and the first student model are selected next time, a model pair with better performance is selected for further evolution and distillation, thereby improving the efficiency of obtaining the target model.

[0011] In a possible implementation, sampling from the model pair list can include: collecting, from the model pair list, a model pair with an evaluation result higher than a preset result; obtaining the teacher model and the first student model according to the structure of the model pair.

[0012] In the embodiments of the present application, when the initialized teacher model and the first student model are selected, the model pairs included in the model pair list can be referred to, and a model pair with better performance can be selected for further evolution and distillation, so as to improve the efficiency of obtaining the target model.

[0013] In a possible implementation, the method can further include: constructing a super network, the super network including a plurality of sub-networks, wherein each sub-network includes at least one network layer, and each network layer includes at least one basic unit; training the super network using the training set at least once to obtain a trained super network, i.e., a plurality of trained sub-networks, and the teacher model can be collected from the super network.

[0014] In the embodiments of the present application, a large-scale super network can be trained in advance as a teacher model pool, so that after the structure of the teacher model is known, a trained teacher model can be collected from the super network, thereby improving the efficiency of obtaining the teacher model.

[0015] In a possible implementation, any one of the at least one training can include: collecting at least one sub-network from the super network; and training the at least one sub-network to obtain the trained super network.

[0016] Therefore, in the embodiments of the present application, during the training of the super network, the sub-networks in the super network can be trained, so that each part of the sub-networks in the super network can have better output.

[0017] In a possible implementation, the at least one sampling from the super network can include: collecting a structure parameter from a structure sampling space, the structure sampling space including one or more of the following: a depth range, a width range, or a convolution kernel size range, the width range being a range of the number of input channels and output channels of the basic units included in the sampled sub-network, the depth range being a range of the number of network layers in the sampled sub-network, and the convolution kernel size range including a range of the size of the convolution kernel in the sampled sub-network; and collecting a sub-network from the super network according to the structure parameter.

[0018] Therefore, in the embodiments of the present application, sampling can be performed from the dimensions of depth, width, or convolution kernel size, so that the sub-networks in the super network can be trained from different dimensions, the training of the super network can be more comprehensively completed, and a super network with higher output precision can be obtained.

[0019] In a possible implementation, the types of the structure parameters sampled from the structure sampling space in each training process of the at least one iteration training are not more than two; the at least one subnetwork includes a first subnetwork and a second subnetwork, the structure of the first subnetwork is larger than that of the second subnetwork, and the first subnetwork is trained before the second subnetwork, which can also be understood as that the structure of the subnetwork sampled for the first time is larger than that of the subnetwork sampled for the second time, and the subnetwork sampled for the first time is trained before the subnetwork sampled for the second time.

[0020] Therefore, in the implementation of the present application, the subnetworks can be sampled in a progressive shrinking manner to gradually complete the training of the subnetworks in the super network.

[0021] In a possible implementation, in the process of evolving the first student model at least once, the parameters of the network layer with the changed structure in the evolved first student model are obtained by inheriting the parameters of the corresponding network layer in the first student model before evolution.

[0022] Therefore, in the implementation of the present application, the network layer with the changed structure in the evolved first student model can inherit the parameters of the network layer before the change, and the subsequent distillation can be performed on the basis of the more optimal parameters, thereby improving the convergence speed of the first student model.

[0023] In a possible implementation, the preset condition includes one or more of the following: the output accuracy is greater than a first threshold, or the average accuracy is greater than a second threshold, or the loss value is not greater than a third threshold, or the inference time length is not greater than a fourth threshold, or the number of floating-point operations (FLOPs) is not greater than a fifth threshold, or the comprehensive score is greater than a sixth threshold, the average accuracy is an average of a plurality of accuracies obtained by evaluating the updated first student model multiple times, the inference time length is a time length for obtaining an output result from the neural network according to an input, and the comprehensive score is obtained by performing weighted operation on at least two of the output accuracy, the average accuracy, the loss value, the inference time length, or the FLOPs.

[0024] In a possible implementation, the first student model is used to perform a target detection task.

[0025] In a possible implementation, distilling the evolved first student model using the teacher model can include: distilling the evolved first student model using the teacher model through a training set to obtain an updated first student model, the training set including at least one image sample, and the updated first student model being used to perform at least one of a target detection task, a classification task, or a segmentation task.

[0026] Therefore, in the embodiments of the present application, the image sample can be used for distillation to obtain the updated first student model, so that the updated student model can be used for image processing tasks such as target detection tasks, classification tasks or segmentation tasks, and is suitable for various image processing scenes.

[0027] In a second aspect, the present application provides a target detection method, comprising: obtaining an input image; inputting the input image into a target model to output information of an object included in the input image, the target model being an updated student model obtained by using a teacher model to distill a student model, wherein at least one evolution is performed on the student model in the process of using the teacher model to distill the student model to obtain an evolved student model, the teacher model is used to distill the evolved student model to obtain the updated student model, and any one of the at least one evolution includes transforming a structure of the student model according to an evolution action to obtain the evolved student model, the evolution action can include at least one of adding a layer, layer rearrangement or pruning, the added layer indicates adding at least one network layer in the student model, the layer rearrangement indicates rearranging at least one network layer in the student model, and the pruning indicates reducing the structure of the student model.

[0028] Therefore, in the embodiments of the present application, the structure search, pruning and distillation are combined to quickly deform the teacher model or the student model, improve the efficiency of changing the model, and use the trained teacher model to distill the student model, quickly complete the training of the student model, and apply the student model in image processing to output the information of the object included in the input image.

[0029] In a possible implementation, the information of the object included in the input image includes: a category of the object and / or a position of the object in the input image.

[0030] In a possible implementation, if the updated first student model does not meet the preset condition, a second student model is obtained, at least one evolution is performed on the second student model, and the teacher model or a new teacher model is used to distill the evolved second student model to obtain a second student model updated again.

[0031] Therefore, in the embodiments of the present application, if the student model meeting the preset condition is not obtained, the steps of evolution and distillation can be repeatedly performed until the student model meeting the preset condition is obtained.

[0032] In a possible implementation, the obtaining of the teacher model and the first student model can include: obtaining the teacher model and the first student model according to a model pair list, the model pair list including at least one model pair, each model pair including a structure of a teacher model and a structure of a first student model.

[0033] In the embodiments of the present application, the initial teacher model and the student model can be selected by setting the model pair list. Thus, a model pair with better performance can be selected from the model pair list as the initial teacher model pair, so as to improve the convergence speed of the student model and improve the efficiency of obtaining the student model meeting the preset condition.

[0034] In a possible implementation, the method can further include: if the updated first student model does not meet the preset condition, adding the teacher model and the updated first student model as a model pair to the model pair list.

[0035] In the embodiments of the present application, if the distilled first student model does not meet the preset condition, the structure of the teacher model and the distilled first student model is added to the model pair list, so that the next time the initial teacher model and the first student model are selected, a model pair with better performance can be selected from the model pairs included in the model pair list for further evolution and distillation, thereby improving the efficiency of obtaining the target model.

[0036] In a possible implementation, the sampling from the model pair list can include: collecting, from the model pair list, a model pair with an evaluation result higher than a preset result; and obtaining the teacher model and the first student model according to the structure of the model pair.

[0037] In the embodiments of the present application, when the initial teacher model and the first student model are selected, a model pair with better performance can be selected from the model pairs included in the model pair list for further evolution and distillation, thereby improving the efficiency of obtaining the target model.

[0038] In a possible implementation, the method can further include: constructing a super network, the super network including a plurality of sub-networks, each sub-network including at least one network layer, and each network layer including at least one basic unit; performing at least one training on the super network using a training set to obtain a trained super network, i.e., a plurality of trained sub-networks, and the teacher model can be collected from the super network.

[0039] In the embodiments of the present application, a large-scale super network can be trained in advance as a teacher model pool, so that after the structure of the teacher model is known, a trained teacher model can be collected from the super network, thereby improving the efficiency of obtaining the teacher model.

[0040] In a possible implementation, any one of the at least one training can include: collecting at least one subnetwork from the super network; and training the at least one subnetwork to obtain the trained super network.

[0041] Therefore, in the implementation of the present application, during the training of the super network, the subnetworks in the super network can be trained, so that the subnetworks in each part of the super network can have better outputs.

[0042] In a possible implementation, the at least one sampling from the super network can include: collecting a structure parameter from a structure sampling space, the structure sampling space including one or more of the following: a depth range, a width range, or a convolution kernel size range, the width range being a range of the number of input channels and output channels of a basic unit included in the sampled subnetwork, the depth range being a range of the number of network layers in the sampled subnetwork, and the convolution kernel size range including a range of the size of a convolution kernel in the sampled subnetwork; and collecting the subnetwork from the super network according to the structure parameter.

[0043] Therefore, in the implementation of the present application, sampling can be performed from the dimensions of depth, width, or convolution kernel size, so that the subnetworks in the super network are trained from different dimensions, the training of the super network is more comprehensively completed, and the super network with higher output precision is obtained.

[0044] In a possible implementation, the type of the structure parameter collected from the structure sampling space in each training of the at least one iterative training is not more than two; the at least one subnetwork includes a first subnetwork and a second subnetwork, the structure of the first subnetwork is greater than the structure of the second subnetwork, and the first subnetwork is trained before the second subnetwork.

[0045] Therefore, in the implementation of the present application, the subnetworks can be collected in a progressive shrinking manner, and the training of the subnetworks in the super network is gradually completed.

[0046] In a possible implementation, during the at least one evolution of the first student model, the parameters of the network layer with the changed structure in the evolved first student model are obtained by inheriting the parameters of the corresponding network layer in the first student model before evolution.

[0047] Therefore, in the implementation of the present application, the network layer with the changed structure in the evolved first student model can inherit the parameters of the network layer before change, and the subsequent distillation can be performed on the basis of the more optimal parameters, so that the convergence speed of the first student model is improved.

[0048] In a possible implementation, the preset condition comprises one or more of the following: the output accuracy is greater than a first threshold, or the average accuracy is greater than a second threshold, or the loss value is not greater than a third threshold, or the inference time length is not greater than a fourth threshold, or the number of floating-point operations (FLOPs) is not greater than a fifth threshold, or the comprehensive score is greater than a sixth threshold, the average accuracy is an average of a plurality of accuracies obtained by performing a plurality of evaluations on the updated first student model, the inference time length is a time length for obtaining an output result from the neural network according to an input, and the comprehensive score is obtained by performing a weighted operation on at least two of the output accuracy, the average accuracy, the loss value, the inference time length, or the FLOPs.

[0049] In a third aspect, an embodiment of the present application provides a neural network distillation apparatus, which has the function of implementing the neural network distillation method in the first aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0050] In a fourth aspect, an embodiment of the present application provides a target detection apparatus, which has the function of implementing the target detection method in the second aspect. The function can be implemented by hardware, or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0051] In a fifth aspect, an embodiment of the present application provides a neural network distillation apparatus, which includes a processor and a memory, wherein the processor and the memory are interconnected through a circuit, and the processor invokes program code in the memory to perform functions related to processing in the neural network distillation method in any of the first aspects. Optionally, the neural network distillation apparatus can be a chip.

[0052] In a sixth aspect, an embodiment of the present application provides a target detection apparatus, which includes a processor and a memory, wherein the processor and the memory are interconnected through a circuit, and the processor invokes program code in the memory to perform functions related to processing in the target detection method in any of the second aspects. Optionally, the target detection apparatus can be a chip.

[0053] In a seventh aspect, an embodiment of the present application provides a neural network distillation apparatus, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is configured to perform functions related to processing in the first aspect or any of the optional implementation manners of the first aspect.

[0054] In an eighth aspect, an embodiment of the present application provides a target detection apparatus, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit acquires program instructions through the communication interface. The program instructions are executed by the processing unit. The processing unit is configured to perform the processing-related functions in the second aspect or any of the optional implementation forms of the second aspect.

[0055] In a ninth aspect, an embodiment of the present application provides a computer-readable storage medium including instructions, which, when executed on a computer, cause the computer to perform the method in the first aspect or any of the optional implementation forms of the second aspect.

[0056] In a tenth aspect, an embodiment of the present application provides a computer program product including instructions, which, when executed on a computer, cause the computer to perform the method in the first aspect or any of the optional implementation forms of the second aspect. BRIEF DESCRIPTION OF DRAWINGS

[0057] Figure 1 An artificial intelligence subject framework schematic diagram for application of the present application;

[0058] Figure 2 A system architecture schematic diagram provided by the present application;

[0059] Figure 3 A convolutional neural network structure schematic diagram provided by an embodiment of the present application;

[0060] Figure 4 Another convolutional neural network structure schematic diagram provided by an embodiment of the present application;

[0061] Figure 5 Another system architecture schematic diagram provided by the present application;

[0062] Figure 6 A neural network distillation method flowchart schematic diagram provided by an embodiment of the present application;

[0063] Figure 7 Another neural network distillation method flowchart schematic diagram provided by an embodiment of the present application;

[0064] Figure 8 Another neural network distillation method flowchart schematic diagram provided by an embodiment of the present application;

[0065] Figure 9 A super network structure schematic diagram provided by an embodiment of the present application;

[0066] Figure 10 A construction unit structure schematic diagram provided by an embodiment of the present application;

[0067] Figure 11 A structural schematic diagram of a sub-network provided for an embodiment of the present application;

[0068] Figure 12 A flowchart of another neural network distillation method provided for an embodiment of the present application;

[0069] Figure 13 A model evolution mode schematic diagram provided for an embodiment of the present application;

[0070] Figure 14 Another model evolution mode schematic diagram provided for an embodiment of the present application;

[0071] Figure 15 A structural schematic diagram of a student model provided for an embodiment of the present application;

[0072] Figure 16 Another model evolution mode schematic diagram provided for an embodiment of the present application;

[0073] Figure 17 Another model evolution mode schematic diagram provided for an embodiment of the present application;

[0074] Figure 18 Another model evolution mode schematic diagram provided for an embodiment of the present application;

[0075] Figure 19 Another model evolution mode schematic diagram provided for an embodiment of the present application;

[0076] Figure 20 A flowchart of another neural network distillation method provided for an embodiment of the present application;

[0077] Figure 21 A flowchart of a target detection method provided for an embodiment of the present application;

[0078] Figure 22 A student model output effect schematic diagram provided for an embodiment of the present application;

[0079] Figure 23 A structural schematic diagram of a neural network distillation device provided for an embodiment of the present application;

[0080] Figure 24 A structural schematic diagram of a target detection device provided for an embodiment of the present application;

[0081] Figure 25 A structural schematic diagram of another neural network distillation device provided for an embodiment of the present application;

[0082] Figure 26 A structural schematic diagram of a chip provided for an embodiment of the present application. DETAILED DESCRIPTION

[0083] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0084] The neural network distillation method provided in the present application can be applied in an artificial intelligence (AI) scene. AI is a theory, method, technology and application system for using a digital computer or a machine controlled by a digital computer to simulate, extend and expand human intelligence, perceive environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science, which attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. The research in the field of artificial intelligence includes robots, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, AI basic theory, etc.

[0085] Figure 1 An artificial intelligence subject framework is shown, which describes the overall workflow of an artificial intelligence system and is suitable for general artificial intelligence field requirements.

[0086] The above-mentioned artificial intelligence subject framework will be described below from two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0087] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensation process of "data-information-knowledge-wisdom".

[0088] The "IT value chain" reflects the value brought by artificial intelligence to the information technology industry from the underlying infrastructure of human intelligence, information (provision and processing technology implementation) to the industrial ecological process of the system.

[0089] (1) Infrastructure:

[0090] Infrastructure provides computing power support for artificial intelligence systems, realizes communication with the external world, and realizes support through the underlying platform. Communication with the outside world through sensors; computing power is provided by intelligent chips such as central processing units (CPU), neural network processing units (NPU), graphics processing units (GPU), application specific integrated circuits (ASIC), or field programmable gate arrays (FPGA) and other hardware acceleration chips); The underlying platform includes distributed computing frameworks and related platform guarantees and support for networks, which can include cloud storage and computing, interconnection networks, etc. For example, sensors and external communication obtain data, which are provided to intelligent chips in the distributed computing system provided by the underlying platform for computing.

[0091] (2) Data

[0092] The data of the upper layer of the infrastructure is used to represent the data source of the artificial intelligence field. The data relates to graphics, images, speech, video, text, and also relates to Internet of Things data of traditional devices, including business data of existing systems and sensing data such as force, displacement, liquid level, temperature, and humidity.

[0093] (3) Data processing

[0094] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision making, and other methods.

[0095] Among them, machine learning and deep learning can model, extract, preprocess, train, etc. symbolic and formalized intelligent information of data.

[0096] Reasoning refers to the process of simulating human intelligent reasoning methods in computers or intelligent systems, using formalized information to perform machine thinking and solve problems according to reasoning control strategies, and the typical function is search and matching.

[0097] Decision making refers to the process of decision making after intelligent information is reasoned, which usually provides functions such as classification, sorting, and prediction.

[0098] (4) General capabilities

[0099] After the data is processed by the above-mentioned data processing, some general capabilities can be formed based on the results of the data processing, such as algorithms or a general system, for example, translation, text analysis, computer vision processing (such as image recognition, target detection, etc.), speech recognition, etc.

[0100] (5) Intelligent product and industry application

[0101] Intelligent product and industry application refers to the product and application of artificial intelligence system in various fields, which is the packaging of the overall solution of artificial intelligence, and realizes the application of intelligent information decision product. The application fields mainly include intelligent manufacturing, intelligent transportation, smart home, intelligent medical treatment, intelligent security, automatic driving, safe city, intelligent terminal, etc.

[0102] Referring to the accompanying drawings Figure 2 , the embodiment of the present application provides a system architecture 200. The system architecture includes a database 230, a client device 240. The data acquisition device 260 is used to collect data and store it into the database 230, and the training module 202 generates the target model / rule 201 based on the data maintained in the database 230. How the training module 202 obtains the target model / rule 201 based on the data will be described in detail below, and the target model / rule 201 is the neural network trained in the following embodiments of the present application. For details, please refer to the related description in the following Figures 6-22 .

[0103] The computing module can include a training module 202, and the target model / rule output by the training module 202 can be applied in different systems or devices. In the accompanying Figure 2 , the execution device 210 is configured with a transceiver 212, which can be a wireless transceiver, an optical transceiver or a wired interface (such as an I / O interface) and the like, and interacts with external devices. The "user" can input data to the transceiver 212 through the client device 240, for example, in the following embodiments of the present application, the client device 240 can send a target task to the execution device 210, request the execution device to build a neural network, and send a database for training to the execution device 210.

[0104] The execution device 210 can call data, code, etc. in the data storage system 250, or store data, instructions, etc. in the data storage system 250.

[0105] The calculation module 211 processes the input data using the target model / rule 201. Specifically, the calculation module 211 is used to: first, obtain the teacher model and the student model; evolve the student model to obtain the evolved student model, wherein the evolution includes: transforming the structure of the student model according to the evolution action to obtain the evolved student model, the evolution action includes at least one of adding a layer, layer rearrangement, or pruning, adding a layer indicates adding at least one network layer in the student model, layer rearrangement indicates rearranging at least one network layer in the student model, and pruning indicates reducing the structure of the student model; and distill the evolved student model using the teacher model to obtain the updated student model.

[0106] Finally, transceiver 212 returns the constructed neural network to client device 240 for deployment in client device 240 or other devices.

[0107] At a deeper level, the training module 202 can obtain corresponding target models / rules 201 based on different data for different tasks, so as to provide users with better results.

[0108] In the appendix Figure 2 In the scenario shown, the data input to the execution device 210 can be determined based on the user's input data. For example, the user can operate on the interface provided by the transceiver 212. Alternatively, the client device 240 can automatically input data to the transceiver 212 and obtain results. If the client device 240 needs user authorization to automatically input data, the user can set appropriate permissions on the client device 240. The user can view the results output by the execution device 210 on the client device 240; the specific presentation format can be display, sound, animation, etc. The client device 240 can also act as a data acquisition terminal, storing the acquired data associated with the target task into the database 230.

[0109] It should be noted that the appendix Figure 2 This is merely an exemplary schematic diagram of a system architecture provided in this application embodiment, and the positional relationships between the devices, components, modules, etc., shown in the diagram do not constitute any limitation. For example, in the appendix... Figure 2 In this context, the data storage system 250 is an external memory relative to the execution device 210. In other scenarios, the data storage system 250 can also be placed within the execution device 210.

[0110] The construction, distillation, or update process mentioned in the present application can be performed by the training module 202. It can be understood that the training process of the neural network, i.e., learning the way of space transformation, more specifically, learning the weight matrix. The purpose of training the neural network is to make the output of the neural network as close to the expected value as possible, so the weight vector of each layer of the neural network can be updated according to the difference between the predicted value of the current network and the expected value (of course, the weight vector can be initialized before the first update, i.e., the parameters of each layer in the deep neural network are pre-configured). For example, if the predicted value of the network is too high, the value of the weight in the weight matrix is adjusted to reduce the predicted value, and through continuous adjustment, until the value output by the neural network is close to or equal to the expected value. Specifically, the difference between the predicted value and the expected value of the neural network can be measured by a loss function or an objective function. Taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, and the training of the neural network can be understood as the process of minimizing the loss. The process of updating the weights of the starting point network and training the serial network in the following embodiments of the present application can refer to this process, which will not be described below.

[0111] As shown in Figure 2 , the target model / rule 201 trained by the training module 202 can be the student model mentioned in the present application. Of course, in the stage of training the teacher model, the target model / rule 201 can also be understood as the trained student model.

[0112] The neural network mentioned in the present application (such as the teacher model or the student model) can include various types, such as a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), a residual network, or other neural networks.

[0113] The neural network mentioned in the present application below can be a neural network obtained by searching for a network structure, rearranging the basic units of the network layer, or pruning the structure in the network layer. Exemplarily, the neural network provided by the present application is exemplarily introduced below taking a convolutional neural network (CNN) as an example.

[0114] A CNN (Convolutional Neural Network) is a deep neural network with a convolutional structure. CNN is a deep learning architecture, which refers to learning at multiple levels of abstraction using machine learning algorithms. As a deep learning architecture, a CNN is a feed-forward artificial neural network where neurons respond to overlapping regions in the input image. A convolutional neural network contains a feature extractor consisting of convolutional layers and subsampling layers. This feature extractor can be viewed as a filter, and the convolution process can be seen as using a trainable filter to convolve with an input image or a convolutional feature map. A convolutional layer is a layer of neurons in a convolutional neural network that performs convolution processing on the input signal. In a convolutional layer of a convolutional neural network, a neuron may only be connected to some of its neighboring neurons. A convolutional layer typically contains several feature planes, each composed of rectangularly arranged neural units. Neural units on the same feature plane share weights, which are the convolutional kernel. Shared weights can be understood as the way image information is extracted regardless of location. The underlying principle is that the statistical information of one part of an image is the same as that of other parts. This means that image information learned in one part can also be used in another part. Therefore, the same learned image information can be used for all locations in an image. Within the same convolutional layer, multiple convolutional kernels can be used to extract different image information. Generally, the more convolutional kernels there are, the richer the image information reflected by the convolutional operation.

[0115] Convolutional kernels can be initialized as matrices of random size, and during the training of a convolutional neural network, they can learn appropriate weights. Furthermore, sharing weights directly reduces the number of connections between layers in the convolutional neural network, while also lowering the risk of overfitting.

[0116] Convolutional neural networks can employ backpropagation (BP) to correct the initial parameters of the model during training, thereby reducing the reconstruction error loss. Specifically, forward propagation of the input signal to the output generates an error loss; this error loss information is then propagated back to update the parameters of the initial super-resolution model, leading to convergence of the error loss. The backpropagation algorithm is an error-loss-driven backpropagation process aimed at obtaining optimal model parameters, such as the weight matrix.

[0117] like Figure 3 As shown, the convolutional neural network (CNN) 100 may include an input layer 110, a convolutional / pooling layer 120, wherein the pooling layer is optional, and a neural network layer 130.

[0118] As Figure 3 indicated, the convolutional / pooling layers 120 can include layers as examples 121-126, in one implementation, the 121 layer is a convolutional layer, the 122 layer is a pooling layer, the 123 layer is a convolutional layer, the 124 layer is a pooling layer, the 125 is a convolutional layer, and the 126 is a pooling layer; in another implementation, the 121 and 122 are convolutional layers, the 123 is a pooling layer, the 124 and 125 are convolutional layers, and the 126 is a pooling layer. That is, the output of a convolutional layer can be used as the input of a subsequent pooling layer, or as the input of another convolutional layer for further convolution operation.

[0119] Taking the convolutional layer 121 as an example, the convolutional layer 121 can include a plurality of convolutional operators, also known as kernels, which act as filters to extract specific information from the input image matrix in image processing. The convolutional operator can be essentially a weight matrix, which is usually predefined. In the process of convolution operation on the image, the weight matrix is usually processed on the input image along the horizontal direction one pixel after another (or two pixels after another, depending on the value of the stride), thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is the same as the depth dimension of the input image, and the weight matrix extends to the entire depth of the input image during the convolution operation. Therefore, convolution with a single weight matrix will produce a single depth dimension of the convolution output, but most cases do not use a single weight matrix, but apply multiple weight matrices of the same dimension. The output of each weight matrix is stacked to form the depth dimension of the convolution image. Different weight matrices can be used to extract different features in the image, such as a weight matrix for extracting image edge information, another weight matrix for extracting specific colors of the image, and another weight matrix for blurring unwanted noise in the image, etc. The multiple weight matrices are of the same dimension, and the feature maps extracted by the multiple weight matrices of the same dimension are also of the same dimension. The extracted multiple feature maps of the same dimension are combined to form the output of the convolution operation.

[0120] Generally, the weight values in the weight matrix need to be obtained through a large amount of training in actual application. The weight values obtained through training form each weight matrix, which can extract information from the input image to help the convolutional neural network 100 make correct predictions.

[0121] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0122] Pooling layer:

[0123] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, i.e., ... Figure 3 Layers 121-126 in example 120 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. In image processing, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers can include average pooling and / or max pooling operators to sample the input image to obtain a smaller image size. Average pooling calculates the average value of pixel values ​​within a specific range. Max pooling takes the pixel with the largest value within a specific range as the result of max pooling. Furthermore, similar to how the size of the weight matrix in a convolutional layer should be related to the image size, the operators in a pooling layer should also be related to the image size. The size of the output image after pooling can be smaller than the size of the input pooling layer image. Each pixel in the output image represents the average or maximum value of the corresponding sub-region of the input pooling layer image.

[0124] Neural network layer 130:

[0125] After processing by the convolutional / pooling layers 120, the convolutional neural network 100 is still insufficient to output the required information. As mentioned earlier, the convolutional / pooling layers 120 only extract features and reduce the parameters introduced by the input image. However, to generate the final output information (the required class information or other relevant information), the convolutional neural network 100 needs to utilize neural network layers 130 to generate one or more outputs representing the required number of classes. Therefore, neural network layers 130 may include multiple hidden layers (such as...). Figure 3 The diagram shows layers 131, 132 to 13n, and an output layer 140. In this application, the convolutional neural network is obtained by performing at least one transformation on a selected starting network to obtain a serial network, and then using the trained serial network to obtain the desired convolutional neural network. This convolutional neural network can be used for object detection, image recognition, or image classification, etc.

[0126] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross-entropy, which is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 is completed (as shown in FIG. 1B), the back propagation (as shown in FIG. 1C) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the result output by the output layer of the convolutional neural network 100 and the ideal result. Figure 3 The propagation from 110 to 140 is forward propagation) is completed, the back propagation (as shown in FIG. 1C, the propagation from 140 to 110 is back propagation) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the result output by the output layer of the convolutional neural network 100 and the ideal result. Figure 3 The propagation from 110 to 140 is forward propagation) is completed, the back propagation (as shown in FIG. 1C, the propagation from 140 to 110 is back propagation) will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100 and the error between the result output by the output layer of the convolutional neural network 100 and the ideal result.

[0127] It should be noted that the convolutional neural network 100 as shown in FIG. 1A is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models, for example, as shown in FIG. 1D, multiple convolutional layers / pooling layers are parallel, and the features extracted respectively are all input to the full neural network layer 130 for processing. Figure 3 Figure 4 It should be noted that the convolutional neural network 100 as shown in FIG. 1A is only an example of a convolutional neural network, and in specific applications, the convolutional neural network can also exist in the form of other network models, for example, as shown in FIG. 1D, multiple convolutional layers / pooling layers are parallel, and the features extracted respectively are all input to the full neural network layer 130 for processing.

[0128] Referring to FIG. 3, the embodiments of the present application also provide a system architecture 300. The execution device 210 is implemented by one or more servers, which can be optionally combined with other computing devices, such as data storage, router, load balancer, etc. The execution device 210 can be arranged on one physical site or distributed on multiple physical sites. The execution device 210 can use the data in the data storage system 250 or call the program code in the data storage system 250 to implement the steps of the corresponding neural network distillation method below. Figure 5 Figures 6-21 The user can operate the respective user devices (such as local device 301 and local device 302) to interact with the execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other types of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0129] The user can operate the respective user devices (such as local device 301 and local device 302) to interact with the execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smartphone, a tablet computer, a smart camera, a smart car or other types of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0130] ​​The local device of each user can interact with the execution device 210 through a communication network of any communication mechanism / communication standard, which can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof. Specifically, the communication network can include a wireless network, a wired network, or a combination of a wireless network and a wired network, etc. The wireless network includes but is not limited to any one or more of the following: a 5th-Generation (5G) system, a long term evolution (LTE) system, a global system for mobile communication (GSM) or a code division multiple access (CDMA) network, a wideband code division multiple access (WCDMA) network, wireless fidelity (WiFi), Bluetooth, Zigbee, radio frequency identification (RFID), long range (Lora) wireless communication, near field communication (NFC), etc. The wired network can include an optical fiber communication network or a network composed of coaxial cables, etc.

[0131] In another implementation, one aspect or multiple aspects of the execution device 210 can be implemented by each local device, for example, the local device 301 can provide local data or feedback calculation results for the execution device 210.

[0132] It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 301 implements the functions of the execution device 210 and provides services for its own user, or provides services for the user of the local device 302.

[0133] Based on the foregoing Figures 1-5 Based on the provided system architecture or neural network, the neural network distillation method provided in the present application will be described in detail below.

[0134] First, for the convenience of understanding, some terms involved in the embodiments of the present application are explained.

[0135] Backbone: A network structure used in detectors, segmenters, or classifiers to extract features from input information. Typically, in neural networks, in addition to the backbone, other functional networks may be included, such as region proposal networks (RPNs) and feature pyramid networks (FPNs), to further process the features extracted by the backbone, such as feature classification and semantic segmentation.

[0136] Residual Networks (ResNet): A type of convolutional neural network. In image recognition, they offer higher accuracy compared to traditional convolutional neural networks. Residual network design involves many structurally identical sub-modules, typically connected by a number representing the number of repetitions of each sub-module, such as ResNet50 indicating 50 sub-modules. The backbone network of the student or teacher model mentioned in this application can be considered a residual network.

[0137] Basic Block: A basic block is typically composed of convolutional layers, or it can be understood as a convolutional module. For example, a basic block can be a convolutional module consisting of one or more convolutional kernels. Basic blocks can be categorized into basic blocks (BB), bottleneck blocks (BNB), and bottleneck grouping (ResNeXt) units, etc.

[0138] Average precision (AP): A common metric for evaluating the performance of a detection model. It is calculated as the average of the precision values ​​of multiple outputs of the model.

[0139] Model floating-point operands (FLOPs): The number of floating-point operations a model performs when processing an image of a specified size. It can be used to measure the complexity of a model.

[0140] Model inference time (FPS): The number of image frames that the model can process per second in single-card mode.

[0141] Secondly, the neural network distillation method provided in this application can be applied to various scenarios of building and training neural networks. The following examples illustrate the application scenarios of the neural network distillation method provided in this application, but are not intended to limit it.

[0142] Scenario 1: Target detection in the perception system of an autonomous vehicle

[0143] Target detection in autonomous vehicle perception systems includes pedestrian, vehicle, traffic sign, and lane detection. Besides requiring high accuracy to ensure driving safety, the detection network also needs a sufficiently fast response speed to ensure precise vehicle control. Similarly, for embedded system deployments, small model size and high energy efficiency are also required. The method provided in this application can perform end-to-end searching for high-precision and high-efficiency neural network models based on specific detection data and the network requirements of the platform to be deployed.

[0144] Scenario 2: Cloud Platform Detection of Everything

[0145] Object detection is the most in-demand service on cloud platforms. Faced with newly submitted business datasets or updated business data, it is necessary to quickly search for efficient network structures to complete business delivery.

[0146] See Figure 6 The flowchart of a neural network distillation method provided in this application is as follows.

[0147] 601. Obtain the teacher model and student model.

[0148] The teacher model's output accuracy is higher than that of the student model. For ease of distinction, the student model before evolution will be referred to as the initial model (or the first student model).

[0149] The teacher model and the initial model can be pre-defined network structures or model structures selected from a list of model pairs.

[0150] In one possible implementation, a supernetwork can be constructed, which may include one or more subnetworks. Each subnetwork includes one or more network layers, and each network layer consists of one or more basic units. The supernetwork is a trained network. The teacher model may be a subnetwork extracted from the supernetwork, or the teacher model may be the supernetwork itself.

[0151] Alternatively, when training the supernetwork, the entire supernetwork can be trained directly using the sample set. To further improve the output accuracy of the supernetwork, subnetworks in the supernetwork can also be collected and trained to obtain the trained supernetwork.

[0152] When sampling subnetworks from a supernetwork, sampling can be performed from various dimensions.

[0153] Specifically, structural parameters can be collected from the structural sampling space, and then corresponding sub-networks can be extracted from the supernetwork based on the structural parameters and trained iteratively. This structural sampling space can include various structural parameters, such as depth range, width range, and kernel size. The depth range refers to the depth of the sampled subnetwork, i.e., the range of the number of network layers; the width range refers to the width of the sampled subnetwork, i.e., the range of the number of input and output channels of the basic unit; and the kernel size range refers to the range of the size of the convolutional kernels in the sampled subnetwork. Therefore, in this embodiment, subnetworks can be sampled from various dimensions, fully capturing subnetworks of each dimension, so that subnetworks of various depths, widths, and kernel sizes can be trained, improving the output accuracy of the supernetwork.

[0154] Alternatively, a progressive shrinking approach can be used when training the supernetwork. Specifically, in each process of sampling a subnetwork from the supernetwork for training, a progressive shrinking approach can be adopted, starting training from the supernetwork itself, that is, first sampling the entire supernetwork, and then gradually sampling smaller subnetworks in each dimension for training.

[0155] For example, training a hypernetwork can be divided into multiple stages, with each stage's corresponding structure sampling space including a type of structural parameter. For instance, stage one can directly train the entire hypernetwork; stage two's structure sampling space can include a depth range; stage three's can include a width range; stage four's can include both depth and width ranges; and stage five's can include a range of convolutional kernel sizes. At each stage, corresponding structural parameters can be sampled from the structure sampling space, and then subnetworks can be extracted from the hypernetwork based on these parameters. For example, in stage two, if the depth sampled from the structure search space is 5-10, then layers 5-10 of the hypernetwork can be sampled as subnetworks. Between each stage, smaller subnetworks can be gradually sampled from the hypernetwork for training. For instance, the first training stage can train the entire hypernetwork, and subsequent stages can gradually train smaller subnetworks until all sizes of subnetworks from the hypernetwork are covered. For example, the first training stage precedes the second, and the structure of the subnetwork sampled in the first training stage is larger than that of the subnetwork sampled in the second training stage. This can be understood as follows: at least one subnetwork includes a first subnetwork and a second subnetwork, the structure of the first subnetwork is larger than that of the second subnetwork, and the first subnetwork is trained before the second subnetwork.

[0156] Therefore, in this embodiment of the application, by training the supernetwork and its subnetworks, a model library including a large number of networks is obtained to support the search for teacher models and obtain teacher models with high output accuracy.

[0157] In one possible implementation, the structures of the teacher model and the initial model can be obtained from a list of model pairs. The teacher model can be extracted from the hypernetwork based on the structure of the teacher model obtained from the list, and the initial model can be constructed based on the structure of the student model obtained from the list, or retrieved from a database. This list of model pairs can include multiple model pairs arranged according to evaluation results. Essentially, after each evolution and distillation of the teacher and student models, the teacher and student models can be treated as a model pair, and their structures added to the list. This allows for the selection of the model pair with the better evaluation results as the initial teacher and student models when updating them. The evaluation results of the model pairs can be obtained by assessing the output accuracy, inference time, average accuracy, etc., of the teacher and / or student models. This allows for further evolution and distillation based on the model pair with better output performance, accelerating the convergence of the student model and obtaining a student model that meets preset conditions. This is equivalent to further evolving and distilling based on model pairs with high output quality. Compared to evolving and distilling based on model pairs with poor output quality, this embodiment can obviously improve the convergence speed of the model, quickly obtain a model with better output results, and improve the efficiency of obtaining student models that meet the preset conditions.

[0158] It should be noted that before obtaining the initial teacher and student models for the first time, the model pairs in the model pair list can be the initial model pairs. At this time, a bunch of model pairs can be randomly selected from the model pairs as the structure of the initial teacher and student models. Alternatively, a student model can be randomly constructed, and then a teacher model with a similar structure to the student model can be collected from the hypernetwork as the initial teacher model, etc. The specific adjustments can be made according to the actual application scenario.

[0159] Furthermore, the model pair list can be a list of the structures of the trained model pairs after training multiple model pairs without producing a student model that meets the preset conditions. For example, if two model pairs are trained without producing a student model that meets the preset conditions, the structure or index of those two model pairs can be added to the model pair list. This allows for the direct sampling of model pairs with better output performance from this list for further evolution and training, improving the training efficiency of the student model and quickly obtaining a student model with better output performance.

[0160] 602. Evolve the student model and / or teacher model.

[0161] After obtaining the teacher model and the initial model, the initial model and / or the teacher model can be evolved. The evolution can be performed once or multiple times to obtain the evolved teacher model and / or the evolved student model.

[0162] Specifically, the evolution can be achieved by changing the depth, width, and / or kernel size of the initial model or teacher model, thereby obtaining the evolved student model or the evolved teacher model.

[0163] The evolution of the initial model and the teacher model will be described below.

[0164] I. Evolution of the initial model

[0165] The evolution of the initial model can take various forms, such as adding layers, layer rearrangement, or pruning. Optionally, an action space can be constructed, which can include evolution actions such as adding layers, layer rearrangement, or pruning. Evolution actions are then collected from this action space, and the initial model is evolved based on these collected actions. Adding a layer means adding at least one network layer to the initial model; layer rearrangement means rearranging the order of network layers in the initial model, such as swapping the positions of two layers; pruning means reducing the structure of the initial model by deleting connections between basic units in the network layers, deleting basic units, or reducing the width of basic units.

[0166] Adding layers can increase the complexity of a model. For example, when the initial model structure is too simple or too small, one or more network layers can be added to the initial model to increase its structural size.

[0167] Layer rearrangement allows for the swapping of different stages of a model, thereby achieving a more efficient allocation of computational resources. For example, the positions of two adjacent network layers can be interchanged.

[0168] Pruning can reduce the structure of the initial model, thereby reducing its complexity. For example, if the initial model is too large, pruning can be triggered to remove connections between basic units, remove network layers, remove basic units, or reduce the width of basic units, thus reducing the structure of the initial model.

[0169] In some commonly used neural network pruning methods, a pre-trained model with a manually specified structure is relied upon, and this model often has a fixed structure. However, in the implementation method of this application, structure search and pruning can be combined to quickly change the structure of the initial model, achieving the goal of combining pruning and structure search, and rapidly obtaining initial models with different structures, thereby improving the efficiency of obtaining the evolved student model.

[0170] Furthermore, in the evolved student model, the parameters of the modified network layers can inherit the parameters of the corresponding network layers before the modification. Therefore, when evolving the student model, the new model can inherit most of the parameters from the previous generation model. Parameter inheritance helps the model utilize existing training results during the search process and effectively avoids dependence on pre-training. The new model with parameter inheritance can achieve performance evaluation with minimal training time, thus greatly improving the search efficiency of the student model.

[0171] II. Evolution of the Teacher Model

[0172] The teacher model can be evolved by changing its depth, width, or convolutional kernel size. For example, the number of network layers in the teacher model can be increased, the width of the basic units in the network layers can be decreased, or the size of the convolutional kernels can be adjusted to obtain the evolved teacher model.

[0173] For example, an action evolution space for the teacher model can be constructed, such as operations like increasing depth, increasing width, decreasing depth, decreasing width, or changing the size of the convolution sum. Then, based on this operation and the teacher model, a new teacher model, i.e., the evolved teacher model, can be collected from the hypernetwork.

[0174] Furthermore, the teacher model and the initial model can be evolved simultaneously to obtain evolved teacher and student models. Therefore, new models can be derived from existing models through evolution, combining operations such as structure search and pruning to obtain models with richer structures. This can be understood as follows: by simply evolving the teacher or student model, new teacher and student models can be quickly obtained, improving the search efficiency for teacher and student models and reducing the subsequent evaluation overhead.

[0175] 603. Distill the evolved student model using the teacher model to obtain the updated student model.

[0176] If the teacher model has been evolved, the evolved teacher model is used as the new teacher model. If the initial model has been evolved, the evolved initial model is used as the student model; otherwise, the initial model is used as the student model. Since the teacher model can be a model whose output accuracy is higher than that of the student model after training, the student model can be distilled using the teacher model to obtain an updated student model.

[0177] Specifically, during distillation, samples from the training set can be used as inputs to both the teacher and student models. This training set can include multiple image samples, each with a corresponding label. Distillation is based on the feature layer and output layer; that is, the output of the teacher model's feature layer guides the update of the student model's feature layer, and the output of the teacher model's output layer guides the update of the student model. This ensures that the output of the student model's feature layer becomes increasingly closer to the output of the teacher model's intermediate layers, and the output of the student model's output layer also becomes increasingly closer to the output of the teacher model's output layer, thus achieving knowledge distillation.

[0178] 604. Does the updated student model meet the preset conditions? If yes, proceed to step 605; otherwise, proceed to step 601.

[0179] After obtaining the updated student model, the performance of the student model can be evaluated to determine whether the evaluation result meets the preset conditions. If the updated student model meets the preset conditions, the updated student model is output. If the updated student model does not meet the preset conditions, the teacher model and student model (or the second student model) can be obtained again, that is, step 601 is repeated. This is equivalent to evolving the second student model. Then, the teacher model or the new teacher model is used to perform knowledge distillation on the evolved second student model to obtain the updated second student model. Alternatively, the updated first student model or teacher model can be evolved to obtain the new teacher model and student model, that is, step 602 is repeated.

[0180] Continuing to evolve the student model or teacher model can also be understood as obtaining a new teacher model or a new student model.

[0181] The preset conditions may include, but are not limited to, one or more of the following: the output accuracy of the updated student model is greater than the first threshold, or the average accuracy is greater than the second threshold, or the loss value is not greater than the third threshold, or the inference time is not greater than the fourth threshold, the number of floating-point operations (FLOPs) is not greater than the fifth threshold, or the overall score is greater than the sixth threshold, etc., wherein the average accuracy includes the average of multiple output accuracies obtained by evaluating the updated student model multiple times, the inference time is the time it takes to obtain the output result from the updated student model based on the input, and the overall score is obtained by weighting at least two of the output accuracy, average accuracy, loss value, inference time, or FLOPs.

[0182] Therefore, in the embodiments of this application, student models can be evaluated from various dimensions to select student models with better output results.

[0183] Furthermore, if the updated student model does not meet the preset conditions, the teacher model and the updated student model can be treated as a pair of models, and the structure of this pair can be added to the model pair list. This allows the model pair list to be referenced when obtaining new teacher and student models, and the model pair with better evaluation results can be selected from the list for further evolution and distillation. This enables the next evolution and distillation to be based on the teacher and student models with better output, which is equivalent to improving the output quality of the initial teacher and student models, thereby accelerating the convergence of the student model and improving the efficiency of obtaining a student model that meets the preset conditions.

[0184] 605. Use the updated student model as the target model for performing the preset task.

[0185] If the updated student model meets the preset conditions, such as high output accuracy, high average accuracy, low loss value, or short inference time, then the updated student model can be used as the target model for performing the preset task, that is, the updated student model will be output.

[0186] The preset tasks can include object detection, classification, or segmentation tasks.

[0187] For example, the preset task can be an object detection task, which is to identify all objects of interest (i.e., foreground) in an image and determine the category and location of the foreground. When acquiring the teacher model and the initial model, the teacher model and the initial model can be selected to perform the object detection task.

[0188] Therefore, in this embodiment, methods such as structure search, pruning, and distillation are combined to quickly transform the teacher model or student model, thereby improving the efficiency of changing the model. Furthermore, the trained teacher model is used to perform knowledge distillation on the student model, which quickly completes the training of the student model and thus improves the efficiency of obtaining the optimal output model.

[0189] Furthermore, in the embodiments of this application, by introducing a list of model pairs, the model pair with better output results can be selected as the initial teacher model and student model for further evolution and distillation, which can improve the convergence speed of the model, quickly obtain the student model with better output results, and improve the efficiency of obtaining a student model that meets the preset conditions.

[0190] Furthermore, in the embodiments of this application, lightweight student models can be obtained in various scenarios by combining structure search, pruning, and distillation. Compared to manual design, the method provided in this application can obtain optimal models in various scenarios, has strong generalization ability, and reduces manual costs. Specifically, structure search and pruning can quickly scale the initial model to obtain new student models with different structures. Distillation can quickly improve the output accuracy of the student model, thereby obtaining the optimal student model. This can be understood as using structure search and pruning to find the computationally optimal model, such as a lightweight model, and then using distillation to improve the output performance of the lightweight model, resulting in better computational resource consumption and output accuracy when running the model.

[0191] The method provided in this application can obtain the optimal model for scenarios such as object detection, classification, or image segmentation. For example, when a student model is needed to perform an object detection task, methods such as structure search, pruning, and distillation can be combined to find a model that meets the user's needs. For instance, if the user's requirement is to build a model with high output accuracy, a large number of models can be obtained through structure search, pruning, and distillation, and then the model with the highest output accuracy can be selected. If the user's requirement is lightweight design, a model with a smaller structure and higher output accuracy can be selected, thereby meeting the user's needs while improving the model's output performance.

[0192] The foregoing has introduced the process of the neural network distillation method provided in this application. To further facilitate understanding, the neural network distillation method provided in this application will be described in more detail below in conjunction with specific application scenarios.

[0193] For example, let's take a preset task as the target detection task.

[0194] First, the neural network distillation method provided in this application can be divided into two parts, such as... Figure 7 As shown, one part involves constructing a teacher model pool 71, from which teacher models can be collected. The other part involves the joint optimization of student model search, pruning, and distillation 72, outputting the final student model. The two parts are described below with examples. (See reference...) Figure 7 This application provides a schematic flowchart of another neural network distillation method.

[0195] Part 1: Constructing a Teacher Model Pool

[0196] The construction of the teacher model pool can be divided into several steps, specifically including: first, preparing task data; then, setting the relevant parameters of the teacher model pool for constructing or training the supernetwork; then, training the supernetwork; subsequently, training the subnetworks in the supernetwork using a strategy of progressively shrinking depth; and finally, training the subnetworks in the supernetwork using a strategy of progressively shrinking width, outputting the trained supernetwork, i.e., the teacher model pool. The detailed steps are described below.

[0197] See Figure 8 This application provides a flowchart illustrating the process of constructing a teacher model pool.

[0198] 801. Prepare task data.

[0199] The task data may include the tasks that the models in the teacher model pool need to perform, such as object detection and classification tasks, and may also include the training set used to train the models in the teacher model pool, such as multiple samples and their corresponding labels.

[0200] 802. Set the relevant parameters.

[0201] The relevant parameters can include those involved in training the teacher model pool. Specifically, these parameters can include: the input size sampling space, which can include the range of input image sizes for the sub-networks included in the teacher model pool, where the input size can be understood as the hyperparameters of the networks included in the teacher model pool; the structure sampling space, which can include the structure parameters used when training the super-network or its sub-networks, such as depth range, width range, or kernel size; and the training hyperparameters for each stage. The training process of the teacher model pool can be divided into multiple stages, and the training hyperparameters for each stage can be the same or different. These training hyperparameters can include the learning rate or training duration for each stage. Generally, the larger the various sampling spaces, the greater the corresponding learning rate or training duration.

[0202] Of course, these parameters can also be set synchronously with the training teacher model pool, that is, there is no need to execute step 802 in advance. For example, the corresponding training hyperparameters can be set for each stage before each stage of training, so that the training hyperparameters for each stage do not need to be set before training the hypernetwork.

[0203] In this embodiment, the training of the supernetwork is exemplaryly divided into three stages: training the supernetwork, progressively shrinking the subnetworks at different depths, and progressively shrinking the subnetworks at different widths. That is, the entire supernetwork is trained first, followed by training subnetworks of different depths and widths. Alternatively, subnetworks of different widths can be trained first, followed by subnetworks of different depths; the specific approach can be adjusted according to the actual application scenario. The different stages are described in detail below.

[0204] 803. Training a Supernetwork

[0205] A hypernetwork is a model that includes multiple subnetworks; it can be understood as a large network. The initial hypernetwork can be a constructed network, a network found within a pre-defined search space, or a network constructed based on a network found within a pre-defined search space. Therefore, in this embodiment, the limitation of the search space on the initial hypernetwork can be removed, making it applicable to more flexible scenarios and exhibiting strong generalization ability.

[0206] When training the supernetwork, a multi-scale training strategy can be employed. The input size of the supernetwork can be obtained by sampling from the input size constructed in step 802. Then, the supernetwork is trained using input images of different sizes to obtain the trained supernetwork. It is understandable that when using input images of different sizes as input to the supernetwork...

[0207] A hypernetwork can include one or more building blocks, which are stacked to form the initial hypernetwork. The structure of each building block can be the same or different. It can be understood that the hypernetwork can include multiple nodes, which are connected by at least one basic operation. This basic operation can also be called an operation or operator. Typically, a hypernetwork can include at least one building block, and each building block can include multiple nodes, which are connected by at least one basic operation. For example, a hypernetwork can be composed of multiple building blocks, each of which can have 3, 4, or 5 nodes, etc., connected by basic operations.

[0208] Specifically, the basic operations for connecting nodes can include convolution, pooling, or a combination of convolution and pooling. Examples include mean pooling (avg_pool_3x3) with a 3×3 kernel, max pooling (max_pool_3x3) with a 3×3 kernel, separating convolution (sep_conv_3x3) with a 3×3 kernel, separating convolution (sep_conv_5x5) with a 5×5 kernel, dilated convolution (dil_conv_3x3) with a 3×3 kernel and a dilation rate of 2, dilated convolution (dil_conv_5x5) with a 5×5 kernel and a dilation rate of 2, skip-connect operations, and zeroing operations (setting all neurons at the corresponding position to zero, abbreviated as Zero).

[0209] Taking any two adjacent nodes as an example, these two nodes can be called the first node and the second node, or the upstream node and the downstream node. The output of each basic operation connecting the first node and the second node serves as the input of the second node. The output of the first node, after being processed by each basic operation connecting the first node and the second node, becomes the output of that basic operation. For example, node 0 and node 1 can be connected through five basic operations. The output of node 0, after being processed by these five operations, becomes the input of node 1.

[0210] For example, this supernetwork can be like Figure 9 As shown, the hypernetwork can include one or more building blocks, which are stacked to form the initial hypernetwork. The structure of each building block can be the same or different. Each building block can include one or more nodes, and the building blocks or the nodes within each building block can be connected through one or more basic operations. Taking one building block as an example, its structure can be as follows: Figure 10 As shown, the building block can include 4 nodes, and each node can be connected to the others through 3 basic operations.

[0211] The process of training the supernetwork involves using the training set prepared in step 801 to train the entire supernetwork, thereby obtaining the trained supernetwork. For example, the back propagation (BP) algorithm can be used to correct the initial parameter values ​​in the supernetwork during training, making the loss of the supernetwork smaller and smaller.

[0212] 804. Deep progressive shrinkage training subnetwork.

[0213] In step 802, a depth sampling space, or depth range, can be constructed. Then, depth values ​​are sampled from this depth range. Subnetworks are determined from the supernetwork based on the sampled depth values, and the other parts of the supernetwork are fixed. The subnetwork is then trained. For example, the depth range can include 1-80. A depth value, such as 15, can be sampled from this range. A subnetwork with 15 network layers can then be extracted from the supernetwork for training, and the parts of the supernetwork other than these 15 layers are fixed.

[0214] When determining the different depths of subnetworks to be trained, the width of the subnetwork can be fixed, or a depth value can correspond to multiple different width values ​​to collect subnetworks and train them, or all widths of the subnetwork in the supernetwork can be trained. The specific method can be adjusted according to the actual application scenario.

[0215] Furthermore, when collecting subnetworks of different depths, depth values ​​can be collected from largest to smallest according to the training order. For example, the depth value of the first subnetwork collected is greater than the depth value of the second subnetwork collected. For instance, the depth value of the first sampled subnetwork is 15, and the depth value of the second sampled subnetwork is 10.

[0216] For example, such as Figure 11 As shown, during the Nth training iteration, a subnetwork with a depth of 6 can be used for training, and during the (N+1)th training iteration, a subnetwork with a depth of 4 can be used for training.

[0217] 805. Gradual width contraction training subnetwork.

[0218] Similar to step 804 above, the difference is that the depth mentioned above is replaced by width in this step. This width is the number of input and output channels of the blocks included in the network layer.

[0219] For example, the width range can include 1-15, such as "1" indicating that the output channel of the corresponding basic unit is 1 times the input channel, "2" indicating that the output channel of the corresponding basic unit is 2 times the input channel, and so on.

[0220] Furthermore, when collecting subnetworks of different widths, the width values ​​can be collected from largest to smallest according to the training order. For example, the width value of the first collected subnetwork is greater than the depth value of the second collected subnetwork. For instance, the width value of the first sample is 15, and the width value of the second sample is 10.

[0221] In addition, when collecting subnetworks of each width, a fixed depth value can be used, or multiple subnetworks of different depths can be collected under the same width. The specific method can be adjusted according to the actual application scenario.

[0222] It should be noted that the aforementioned methods of collecting subnetworks from the supernetwork for training from the dimensions of depth and width are merely illustrative examples. Subnetworks can also be collected for training from dimensions such as depth + width, convolutional kernel size, etc. The specific methods can be adjusted according to the actual application scenario, and will not be elaborated here.

[0223] Therefore, in the embodiments of this application, the training of the super network can be divided into multiple stages. Each stage can collect sub-networks from different dimensions for training until the sub-networks of various sizes in the super network are covered, thereby completing the training of the super network and obtaining a large pool of teacher models with high output quality.

[0224] Optionally, after training the teacher model pool, sub-networks can be randomly sampled from the teacher model pool to verify the output performance of the networks in the teacher model pool, thereby ensuring the output accuracy of the teacher model pool. If the output performance of one or more sampled sub-networks is poor, such as output accuracy, average accuracy, or inference time being inferior to the preset values, the teacher model pool can be trained again until a teacher model pool with verification results superior to the preset values ​​is obtained.

[0225] Part Two: Joint Optimization

[0226] After obtaining a large pool of teacher models with high output quality, the teacher models and the initial student models can be selected for subsequent evolution and distillation processes, thereby outputting the distilled student models.

[0227] For example, the steps of joint optimization can be as follows: Figure 12 As shown. Joint optimization can encompass multiple stages, such as... Figure 12 The initialization, evolution, and distillation are shown below. Each stage is described exemplarily below.

[0228] (I) Initialization

[0229] First, the teacher model and student model are initialized to obtain the initial teacher model 1202 and student model 1203.

[0230] To facilitate understanding, the method of obtaining the student model will first be exemplarily introduced. The student model 1203 can be obtained in various ways; some possible methods are described below as examples.

[0231] Method 1: Preset Structure

[0232] The student model 1203 can be a pre-defined fixed structure; for example, the initial structure of the student model can be provided by the user. As mentioned above. Figure 2As shown, the user sends input data to the execution device through the client device 240, and the input data may carry the structure of the student model 1203.

[0233] Method 2: Obtained by searching the search space

[0234] The search space can include a variety of models. When initializing the student model, the student model can be randomly searched from this search space.

[0235] For example, the search space can include the depth range, width range, convolution kernel type, or pooling kernel type required to build the student model. For instance, the depth range can be 10-105, the width range can be 1-16, the convolution kernel type can include 3×3 separating convolution (sep_conv_3x3), 5×5 separating convolution (sep_conv_5x5), 3×3 dilated convolution with a dilation rate of 2 (dil_conv_3x3), 5×5 dilated convolution with a dilation rate of 2 (dil_conv_5x5), etc., and the pooling kernel type can include 3×3 average pooling (avg_pool_3x3), 3×3 max pooling (max_pool_3x3), etc.

[0236] Method 3: Obtain the structure of the student model from the list of model pairs.

[0237] The list of model pairs can include the structures of one or more model pairs, each of which can include a teacher model and a student model. A model pair can be randomly sampled from this list, and the structure of the student model in that pair can be used as the initial structure of the student model in this step. The corresponding subnetwork can then be collected from the hypernetwork according to the structure of the teacher model within that pair, serving as the teacher model.

[0238] Alternatively, model pairs with evaluation results higher than a certain value or with the best evaluation results can be selected from the list of model pairs. The structure of the student model in the model pair can be used as the initialization structure of the student model in this step, and the corresponding subnetwork can be selected from the hypernetwork as the teacher model according to the structure of the teacher model in the pair.

[0239] The model pair list can be formed by adding the teacher model and the distilled student model to the list before the termination condition is met. The model pairs in the list can be arranged according to evaluation results, such as output accuracy, average accuracy, FLOPs, inference time, and Pareto frontier of inference speed for the student model. Therefore, during the next initialization of the teacher and student models, the better-performing model pair (e.g., with evaluation results higher than the preset result) can be selected from the list for further evolution and distillation. This allows the next evolution and distillation to be based on the better-performing initialization model, accelerating the convergence of the student model and improving the efficiency of obtaining a student model that meets the preset conditions.

[0240] In addition, if the current time is the first time to initialize the teacher model and student model, an initial model pair can be generated, which will be used as the structure of the teacher model and student model, and the initial model pair will be added to the model pair list.

[0241] The methods for obtaining the student model have been described above. There are also multiple methods for obtaining the teacher model 1202.

[0242] In one implementation, in the aforementioned method three for obtaining the student model, a model pair can be obtained from the model pair list 1204, and the structure of the student model in the model pair is initialized to the structure of the student model 1203 in this step. Correspondingly, the structure of the teacher model in the model pair is used as the structure of the teacher model 1202 in this step, and the corresponding sub-network that has been trained is extracted from the supernetwork according to the structure as the teacher model 1202.

[0243] Alternatively, a pre-defined model whose evaluation result is higher than a certain threshold can be used as the teacher model 1202. Alternatively, a hypernetwork can be used directly as the teacher model 1202. Alternatively, after initializing the student model, a subnetwork whose structure matches the student model can be searched from the hypernetwork as the teacher model. This matching can include having the same depth, having a certain linear relationship, or having the same width.

[0244] (II) Evolution

[0245] After initializing the teacher model and the student model, the teacher model and / or the student model can be evolved to obtain the evolved teacher model 1205 and / or the evolved student model 1206.

[0246] It should be noted that during evolution, only the student model, only the teacher model, or both models can be evolved simultaneously. For example, pruning or structure search can be performed on the student model to obtain a new student model. If the teacher model remains unchanged, the new student model and teacher model can be combined to form a new model pair. Alternatively, the teacher model can be evolved, and a new subnetwork can be extracted from the teacher model pool 1201 according to the evolved structure to form a new teacher model. This new teacher model and the unevolved student model can also form a new model pair. If both the teacher and student models are evolved, the evolved student model and the evolved teacher model can form a new model pair. It should be noted that in... Figure 12 The example provided is only for the case where both the teacher model and the student model evolve, and is not intended to be limiting. The specific evolution method can be adjusted according to the actual application scenario.

[0247] First, we will introduce how the student model evolved.

[0248] The evolution of the student model can be achieved in various ways. Specifically, it can involve collecting the evolving actions from the action space corresponding to the student model, and then evolving the student model based on these actions to obtain the evolved student model.

[0249] Actions in this action space can include a variety of actions, such as adding layers, rearranging layers, or pruning, etc. For ease of understanding, some actions are described below as examples.

[0250] 1. Add a layer

[0251] Adding a layer means adding a network layer to the student model, or it can also be called an extension.

[0252] For example, the structure of the student model is as follows: Figure 13 As shown, the student model consists of 8 network layers. An additional network layer is added between the 2nd and 3rd layers to obtain the evolved student model. Furthermore, the structure of the added network layer can be similar to that of the 2nd or 3rd layer; here, we take the example where the added network layer's structure is similar to that of the 3rd layer. The parameters of the added network layer can be initialized with the identity matrix, thereby improving the training efficiency of the student model.

[0253] Besides adding layers, basic units can be added to each network layer to change its structure. For example, the structure of a certain network layer in the student model could be as follows: Figure 14As shown, a certain layer of the student model includes block7 and block8. A new block can be added to this layer. The structure of this new block is similar to that of block8. The parameters of the added block are the parameters initialized with the identity matrix, thereby improving the training efficiency of the student model.

[0254] 2. Layered arrangement

[0255] Layer rearrangement refers to the reordering of network layers in a student model, that is, adjusting the order of network layers in the student model. For example, swapping the positions of the first and second network layers, or swapping the positions of blocks within a network layer, etc.

[0256] For example, the structure of a student model can be as follows: Figure 15 As shown, the student model consists of 8 network layers. The positions of the first and second layers can be swapped, with the first layer appearing after the second layer. It should be noted that different network layers in the student model may process feature maps of different scales, and the scales processed by the network layers in the student model are usually arranged according to a certain size, such as... Figure 15 The scale of the network layers shown from left to right can be arranged from large to small. When swapping the positions of network layers, only the positions of network layers with the same processing scale can be swapped, or the step size of the weights of the swapped network layers can be adjusted after swapping network layers with different processing scales. This way, the arrangement of network layers in the student model remains unchanged, and different student models with different structures are obtained while minimizing the negative impact on the student model.

[0257] For example, the structure of a certain network layer in the student model can be as follows: Figure 16 As shown, the network layer includes block2, block3, block4 and block5. The positions of block2 and block3 can be swapped to obtain the network layer after block replay, thus obtaining student models with different structures.

[0258] 3. Pruning

[0259] Pruning refers to reducing the structure of a student model, such as deleting one or more network layers, deleting one or more blocks in a student model network layer, or reducing the input and output channels of one or more blocks.

[0260] In one possible implementation, during pruning, the γ parameter of the BN (batch normalized) layer can be used as the evaluation criterion for the importance of parameters in each layer of the network, and a regularization loss for the γ parameter can be added during training to help the network obtain a more sparsity distribution of the γ parameter: L BN =∑ γ∈Γ|γ|, where γ is a learnable scaling factor for the Batch Normalization (BN) layer, used to measure the importance of each channel, and Γ represents the set of γ parameters for all BN layers in the network. During pruning, channels with small γ values ​​(prune channels) or layers with small average γ values ​​(prune layers) can be deleted based on the proportion of pruning required. Alternatively, the channels, blocks, or layers to be deleted can be randomly determined, and the specific targets can be adjusted according to the actual application scenario.

[0261] For example, the structure of a student model can be as follows: Figure 17 As shown, the student model consists of 8 network layers. The 6th network layer can be removed to obtain the evolved student model, thus reducing the structure of the student model.

[0262] For example, the structure of a certain network layer in the student model can be as follows: Figure 18 As shown, the network layer includes block2, block3, block4 and block5. Block5 can be deleted, thereby changing the structure of the network layer and obtaining one of the network layers in the evolved student model.

[0263] For example, the structure of a block in the student model can be as follows: Figure 19 As shown, before pruning, the block has 3 input channels and 3 output channels. After pruning, the block has 2 input channels and 2 output channels, resulting in the evolved student model.

[0264] In addition to the aforementioned operations of adding layers, rearranging layers, and pruning, the action space can also include other operations, such as changing the type of a block, adding a block in a network layer, or adding input or output channels for a block. The specific operations can be adjusted according to the actual application scenario, and are not limited here.

[0265] Therefore, in this embodiment, structure search and pruning can be combined to modify the structure of the student model, enabling scaling and quickly obtaining a new student model. Furthermore, most actions in the action space support parameter inheritance, meaning the evolved structure can inherit the parameters of the original structure. This facilitates utilizing existing training results during the student model search process and effectively avoids model dependence on pre-training (such as ImageNet pre-training). The new model after parameter inheritance can achieve performance evaluation with minimal training time, thereby significantly improving search efficiency.

[0266] Secondly, the evolution of teacher model 1202 will be introduced below.

[0267] The evolution of the teacher model can take many forms, including changing its depth, width, convolutional kernel size, etc. For example, an action sampling space can be constructed for the teacher model, which can include ways to evolve the teacher model, such as changing its depth or width. After sampling the evolution actions of the teacher model from this action sampling space, the teacher model can be evolved according to these actions to obtain the structure of the evolved teacher model. Then, the subnetworks corresponding to the structure of the evolved teacher model are collected from the supernetwork to obtain the evolved teacher model 1205.

[0268] The difference between the evolution of the teacher model and the student model lies in the fact that the student model is directly evolved from the initial student model to obtain the evolved student model, while the teacher model needs to be evolved from the initial teacher model. Then, based on the structure obtained after the evolution, the evolved teacher model is collected from the hypernetwork to obtain the trained teacher model.

[0269] (III) Distillation

[0270] If both the teacher model and the student model have been evolved, the evolved teacher model is used to perform knowledge distillation on the evolved student model to obtain the distilled student model 1207; if only the teacher model has been evolved, the evolved teacher model is used to perform knowledge distillation on the student model to obtain the updated student model; if only the student model has been evolved, the teacher model can be used to perform knowledge distillation on the evolved student model to obtain the distilled student model 1207.

[0271] Specifically, the distillation method can be based on the task that the student model needs to perform. For example, if the task that the student model needs to perform is related to the foreground in the input image, the student model can be guided based on the foreground-related features or output results of the teacher model; if the task that the student model needs to perform is related to the background of the input image, the student model can be guided based on the background-related features or output results of the teacher model.

[0272] For example, taking the task performed by the student model as the target detection task, we will introduce a possible knowledge distillation method.

[0273] In this embodiment, feature layer distillation and output layer distillation can be used to complete the distillation process. Feature layer distillation uses the output of the teacher model's feature layer to guide the update of the student model's feature layer, while output layer distillation uses the output of the teacher model's output layer to guide the update of the student model. This ensures that the output of the student model's feature layer becomes increasingly closer to the output of the teacher model's intermediate layers, and the output of the student model's output layer also becomes increasingly closer to the output of the teacher model's output layer, thus achieving knowledge distillation.

[0274] The role of feature layer distillation is to help the student network learn the teacher network's feature encoding of the foreground. Specifically, it can use the ROI Align module in a two-stage detector to extract the foreground from the FPN's output features and filter out background noise. During the distillation process, the loss function can be the mean square error (MSE) loss, as shown below:

[0275]

[0276] Among them, F S and F T Let f represent the features of the student model and the teacher model, respectively. adap Let L, W, H, C represent the feature adaptive mapping function, and N represent the feature hierarchy and dimension. e =C×W×H. After calculating the loss value, the student model can be updated in reverse based on the loss value to obtain the updated student model.

[0277] For output layer distillation, cross-entropy loss L can be used. cls To construct the loss value for classification tasks, while for localization tasks, a localization uncertainty L carrying a class attention mechanism can be used. loc To more effectively transfer the teacher network's location information about objects, it can be represented as:

[0278]

[0279]

[0280] Where N represents the number of candidate regions. This represents the score of the predicted category for the i-th candidate region in the teacher network. Let C represent the score of the predicted category of the i-th candidate region in the student network, and let C represent the number of categories. This represents the predicted localization information of the i-th candidate region in the teacher network (e.g., the offset relative to the anchor point in the xywh directions). This represents the confidence level of the current candidate region predicted by the teacher model in class i.

[0281] Therefore, in this embodiment, distillation for object detection can be performed based on foreground-related features or results output by the teacher model. Combined with the aforementioned structure search and pruning operations, the trained student model is obtained through distillation after rapid scaling of the student model, thus improving the efficiency of obtaining the student model. Furthermore, compared to manually designed networks, the neural network distillation method provided in this application has stronger generalization ability, and the output accuracy of the model can improve with training of the teacher model pool, resulting in a student model with higher output accuracy.

[0282] After obtaining the distilled student model, it can be determined whether the distilled student model meets the preset conditions. If the distilled student model meets the preset conditions, it can be used as the target model 1208, that is, the model used to perform the target detection task. If the distilled student model does not meet the preset conditions, the structure of the distilled student model and the structure of the teacher model (if it has been evolved, it will be replaced with the evolved teacher model) can be inserted as a model pair into the model pair list.

[0283] For example, model pairs in the model pair list can be arranged according to a comprehensive score, which can be obtained by weighting at least two of the following: output accuracy, average accuracy, loss value, inference time, or FLOPs of the distilled student model. After obtaining student models that do not meet the preset conditions, a comprehensive score can be applied to these student models, and the models can be inserted into the model pair list according to this comprehensive score. Such a model pair list can be shown in Table 1:

[0284] Model pair Overall score … a+b 92 … c+d 91 … e+f 89 … … … …

[0285] Table 1

[0286] In Table 1, the "Model Pairs" column represents the structure of the model pairs consisting of the student model and the teacher model.

[0287] After obtaining a new model pair, such as g+h, with a comprehensive score of 90, the model pair g+h can be inserted into the list of model pairs, as shown in Table 2:

[0288] Model pair Overall score … a+b 92 … c+d 91 … g+h 90 … e+f 89 … … … …

[0289] Table 2

[0290] During the next initialization step (I), the model pair with the highest overall score or an overall score higher than 90 can be selected from the list of model pairs as the initial teacher model and student model structure. This allows for further evolution and distillation based on the better-performing initial model, which can further improve the output of the initial model and obtain a student model whose output meets the preset conditions.

[0291] The foregoing has provided a detailed description of the neural network distillation method provided in this application. The following example uses a more specific scenario, such as... Figure 20 The architecture of the neural network distillation method provided in this application will be illustrated more vividly.

[0292] The neural network distillation method provided in this application can be divided into two parts, namely: Figure 20 The diagram illustrates the construction of a teacher model pool and hill-climbing joint optimization.

[0293] In constructing the teacher model pool, a supernetwork was first built. Images of different resolutions were used as input to the supernetwork, allowing it to be trained at different scales, resulting in a trained supernetwork. Then, subnetworks within the supernetwork were selected based on depth and depth plus width dimensions. The parameters of the other subnetworks were initially fixed, and these subnetworks were then trained. After training the subnetworks in each dimension of the supernetwork, the final, flexible teacher model pool was obtained.

[0294] During the hill-climbing joint optimization phase, before distillation, sub-networks can be extracted from the teacher model pool as teacher models, such as... Figure 20 The examples T0, T1, and T2 are shown, and corresponding student models are constructed, such as... Figure 20 The S0, S1, and S2 shown form model pairs, such as T0 and S0 forming a model pair, and T1 and S1 forming a model pair. Then, the teacher model is used to distill the student model to obtain the distilled student model.

[0295] Before obtaining a student model that meets the preset conditions, the student model and / or teacher model can be evolved. For example, pruning or other structure search operations can be performed on the student model S0 to obtain the evolved student model S1. S1 and the teacher model T1 form a new model pair. Then, distillation can be performed based on the new model pair to obtain the distilled student model, until a student model that meets the preset conditions is obtained. Finally, the student model that meets the preset conditions is output, such as... Figure 20 The S shown final .

[0296] Therefore, in this embodiment, the hypernetwork is first trained, and then its subnetworks in each dimension are trained separately to obtain a pool of teacher models with good performance. Then, subnetworks are extracted from this pool as teacher models and paired with student models to form model pairs. The teacher models are then used to perform knowledge distillation on the student models to obtain distilled student models. Before obtaining student models that meet preset conditions, the student models can be evolved, such as through pruning or structure search, thereby achieving rapid scaling of the student models and efficiently obtaining student models that meet the preset conditions.

[0297] Furthermore, based on the updated student model output by the neural network distillation method provided in this application, this application also provides a foreground detection method, the process of which is as follows: Figure 21 As shown.

[0298] 2101. Obtain the input image.

[0299] Among them, the target model is deployed in the aforementioned Figure 5 In the execution device 210 shown, the user can send the input image to the execution device through local device 301 or 302. For example, after the execution device outputs the target model, the target model can be sent to local device 301 or 302 to deploy the target model on local device 301 or 302. The user can directly input the input image into local device 301 or 302 to obtain the output result.

[0300] 2102. Take the input image as the input to the target model and output the information of the objects included in the input image.

[0301] The information about the objects included in the input image may specifically include the object's location in the input image (e.g., represented by coordinates) or the object's category.

[0302] Specifically, the target model and its acquisition method can be found in the aforementioned documents. Figures 6-20 The relevant descriptions will not be repeated here.

[0303] Therefore, in this embodiment, methods such as structure search, pruning, and distillation are combined to quickly transform the teacher model or student model. The teacher model with higher output accuracy is used to perform knowledge distillation on the student model to improve the output accuracy of the student model. The student model with better output performance is used as the target model to obtain a target model with higher output accuracy, thereby making the prediction results of the target model for the input image more accurate.

[0304] To facilitate understanding, the output performance of the neural network distillation method provided in this application in a specific application scenario (taking object detection as an example) will be further explained. First, the models in the teacher model pool can be constructed using the ResNet-FPN backbone network, and the detection heads all use the commonly used RCNN heads. Specific relevant parameters may include the following.

[0305] Any model in the flexible teacher pool needs to support operation at multiple input sizes. The input size sampling space can be set to: [512x512, 800x600, 1080x720, 1333x800].

[0306] Considering that the teacher model needs to have high accuracy, a model with a width equivalent to 1.5 times that of RenNet101-FPN can be constructed as a supernetwork.

[0307] Progressive depth training gradually adds shallower subnets to the training space, with the corresponding progressive depth space designed to gradually expand from the initial 101 layers to 25-101 layers. Similarly, the progressive width space is designed to gradually expand from the initial 1.5 times the width to 1-1.5 times the width.

[0308] Then, a sub-network can be selected from the trained teacher model pool as the teacher model, and knowledge distillation can be performed on the constructed student model to obtain the distilled student model.

[0309] For example, regarding the effect of pruning operations, ResNet50-FPN (R50) and ResNet101-FPN (R101) were used as student models. Based on the pre-trained student models, 10%-30% of the parameters were pruned, and the results were compared with the baseline pre-trained model (i.e., the model with a pruning ratio of 0%). The comparison results are shown in Table 3.

[0310]

[0311] Table 3

[0312] Clearly, even with a pruning ratio of 30%, the output performance of the student model did not decrease significantly. This is equivalent to reducing the model structure while maintaining the output performance, resulting in a lighter model.

[0313] For example, regarding the effect of distillation, R18-R50 and R50-R101 were used as student-teacher models respectively to verify the performance. The verification results were compared with the fine-grained feature imitation (FGFI) and task adaptive regularization (TAR) methods of the object detection model. The results are shown in Table 4.

[0314]

[0315] Table 4

[0316] Clearly, the neural network distillation method provided in this application achieves better output results while using the same student model. This is equivalent to selecting the foreground and / or background parts for distillation separately for different tasks, thus achieving superior distillation performance.

[0317] For example, regarding the effectiveness of the progressive search training strategy used when training the teacher model pool provided in this application, several subnets with the same structure as the conventional model (such as R18-FPN, R50-FPN, etc.) were extracted from the elastic teacher pool built on the COCO dataset and compared with the results of training them separately using the normal training strategy. The results are shown in Table 5.

[0318]

[0319] Table 5

[0320] Clearly, the progressive search training strategy provided in this application for training the teacher model pool can significantly improve the output accuracy of the teacher model pool, resulting in better-performing teacher models and consequently improving the output performance of student models. This is equivalent to training the structures in the teacher model pool more equitably through the progressive search training strategy, thereby improving the overall output performance of the models included in the teacher model pool.

[0321] Furthermore, since the teacher model pool contains a large number of high-quality models, it can not only be used as the teacher model pool in this application, but also achieve excellent results when directly performing architecture search based on this model pool. Because the performance of all models within it can be directly evaluated without training, the search efficiency is extremely high. Table 6 shows the search results based on the elastic teacher model pool, demonstrating that it outperforms commonly used NAS detection NAS algorithms such as NAS-FPN and SP-NAS in both search efficiency and the performance of the searched models. Due to its extremely low search overhead, this model pool can also be used to quickly search for optimal models adapted to different device operating conditions, thereby achieving the goal of training once and deploying in multiple locations.

[0322]

[0323] Table 6

[0324] For example, regarding the effects of the joint optimization in this application, see [link / reference]. Figure 22 In this application, the teacher model is extracted from the hypernetwork, which is equivalent to dynamically adjusting the teacher model. Compared with the static teacher model, the output accuracy of the student model increases faster and the final performance is higher.

[0325] Table 7 compares the performance of the search teacher model and the fixed teacher model when the student model is fixed. The joint optimization provided in this application is referred to as Joint-DetNAS.

[0326]

[0327] Table 7

[0328] Clearly, the Joint-DetNAS provided in this application improves output accuracy while reducing student model FLOPs without the need for pre-training. This is equivalent to improving the output accuracy of a lightweight student model, resulting in a student model with superior structure and output accuracy.

[0329] To demonstrate the effectiveness of Joint-DetNAS provided in this application, common object detection models such as ResNet-FPN series were used as the base models for joint optimization. As shown in Table 8, Joint-DetNAS can consistently deliver performance improvements for input models of different sizes, with an average AP improvement of over 2.5. The FLOPs and FPS of the student models are also significantly improved. Table 9 shows a comparison between the optimized model and commonly used object detection models. Commonly used object detection models can employ cascaded two-stage detectors (Cascade Region-based CNN, Cascade RCNN), fully convolutional one-stage object detection (FCOS), composite backbone network (CBNet), backbone search for object detection (DetNAS), structural-to-modular NAS (SM-NAS), neural architecture search-feature pyramid network (NAS-FPN), scalable and efficient object detection (efficientDet), or serial-to-parallel NAS (SP-NAS). Under the same scale conditions, the model obtained by Joint-DetNAS significantly outperforms commonly used manually designed object detection models and object detection models automatically obtained by NAS in both AP and FPS.

[0330]

[0331] Table 8

[0332]

[0333] Table 9

[0334] Regarding search efficiency, Table 10 shows a comparison of the search efficiency of Joint-DetNAS with commonly used object detection NAS algorithms DetNAS, NAS-FPN, and SP-NAS. Clearly, Joint-DetNAS can obtain a higher quality model with a relatively lower search cost. Essentially, the method provided in this application, which adds distillation to structure search and pruning, can obtain a student model with better output performance while reducing the overhead of the search model.

[0335] Search way FLOPs AP Search space Search overhead Random search - - 50 1200 DetNAS 289.4 40.0 1000 70 NAS-FPN 281.3 39.9 10000 >>500 SP-NAS 349.3 41.7 200 200 Joint-DetNAS 145.7 43.9 100 200

[0336] Table 10

[0337] The foregoing has provided a detailed description of the specific process of the method provided in this application. Based on the foregoing method process, the apparatus for executing the method process will be described in detail below.

[0338] First, refer to Figure 23 This application provides a schematic diagram of a neural network distillation apparatus, which can be used to perform the aforementioned... Figures 6-22 The corresponding method's steps, the neural network distillation apparatus may include:

[0339] Module 2301 is used to acquire the teacher model and the first student model;

[0340] Evolution module 2302 is used to evolve the first student model to obtain the evolved first student model. The evolution includes: transforming the structure of the first student model according to the evolution action to obtain the evolved first student model. The evolution action may include at least one of adding a layer, layer rearrangement, or pruning. Adding a layer indicates adding at least one network layer in the first student model. Layer rearrangement indicates rearranging at least one network layer in the first student model. Pruning indicates reducing the structure of the first student model.

[0341] Distillation module 2303 is used to distill the evolved first student model using the teacher model to obtain an updated first student model.

[0342] In one possible implementation, the neural network distillation device may further include: an output module 2304, which is used as the target model if the updated first student model meets the preset conditions, and the target model is used to perform the preset task.

[0343] In one possible implementation, if the updated first student model does not meet the preset conditions, the acquisition module 2301 acquires the second student model, the evolution module 2302 evolves the second student model, and the distillation module 2303 distills the evolved second student model using the teacher model or a new teacher model to obtain the updated second student model.

[0344] In one possible implementation, the acquisition module 2301 is specifically used to sample from the model pair list and obtain the teacher model and the first student model based on the sampling results. The model pair list includes at least one pair of model pairs, and each pair of model pairs includes the structure of a teacher model and the structure of a first student model.

[0345] In one possible implementation, the acquisition module is further configured to add the teacher model and the updated first student model as a model pair to the model pair list if the updated first student model does not meet the preset conditions.

[0346] In one possible implementation, the acquisition module is specifically used to: collect model pairs whose evaluation results are higher than preset results from the model pair list; and obtain the teacher model and the first student model based on the structure of the model pairs.

[0347] In one possible implementation, the neural network distillation apparatus may further include:

[0348] Module 2305 is used to construct a supernetwork, which includes multiple subnetworks;

[0349] Training module 2306 is used to train the supernetwork at least once using the training set to obtain the trained supernetwork. The teacher model is collected from the supernetwork.

[0350] In one possible implementation, any one of the training sessions may include: acquiring at least one subnetwork from the supernetwork; training the at least one subnetwork to obtain the trained supernetwork, i.e., the trained subnetworks.

[0351] In one possible implementation, the training module is specifically used to: acquire structural parameters from a structural sampling space, the structural sampling space including one or more of the following: a depth range, a width range, or a kernel size range, the width range being the range of the number of input and output channels of the basic units included in the sampled subnetwork, the depth range being the range of the number of network layers in the sampled subnetwork, and the kernel size range being the range of the size of the kernels in the sampled subnetwork; and acquire a subnetwork from the supernetwork based on the structural parameters.

[0352] In one possible implementation, the types of structural parameters collected from the structural sampling space in each training process of at least one training session do not exceed two; the at least one sub-network includes a first sub-network and a second sub-network, the structure of the first sub-network being larger than the structure of the second sub-network, and the first sub-network being trained before the second sub-network.

[0353] In one possible implementation, during at least one evolution of the first student model, the parameters of the network layers whose structure has been altered in the evolved first student model are obtained by inheriting the parameters of the corresponding network layers in the first student model before the evolution.

[0354] In one possible implementation, the preset conditions include one or more of the following: output accuracy is greater than a first threshold, or average accuracy is greater than a second threshold, or loss value is not greater than a third threshold, or inference time is not greater than a fourth threshold, or floating-point operations (FLOPs) are not greater than a fifth threshold, or comprehensive score is greater than a sixth threshold. The average accuracy is the average of multiple accuracies obtained by evaluating the updated first student model multiple times, the inference time is the time it takes for the neural network to obtain the output result based on the input, and the comprehensive score is obtained by weighting at least two of the output accuracy, average accuracy, loss value, inference time, or FLOPs.

[0355] In one possible implementation, the student model can be used to perform object detection tasks.

[0356] In one possible implementation, the distillation module 2303 can be specifically used to distill the evolved first student model using the training set through the teacher model to obtain an updated first student model. The updated first student model is used to perform at least one of the following tasks: object detection, classification, or segmentation.

[0357] See Figure 24 This application provides a schematic diagram of a target detection device, which can be used to perform the aforementioned... Figure 21 The corresponding method steps, the target detection device may include:

[0358] Transceiver module 2401 is used to acquire input images;

[0359] Output module 2402 is used to take an input image as input to a target model and output information about the objects included in the input image. The target model is an updated student model obtained by knowledge distillation of the student model using the teacher model. In the process of knowledge distillation of the student model using the teacher model, the student model is evolved to obtain an evolved student model. The evolved student model is then knowledge distilled using the teacher model to obtain an updated student model. The evolution includes: transforming the structure of the student model according to the evolution action to obtain the evolved student model. The evolution action may include at least one of adding a layer, layer rearrangement, or pruning. Adding a layer indicates adding at least one network layer to the student model. Layer rearrangement indicates rearranging at least one network layer in the student model. Pruning indicates reducing the structure of the student model.

[0360] In one possible implementation, the information of the objects included in the input image includes: the category of the object and / or the location of the object in the input image.

[0361] More specifically, the target model can be achieved through the aforementioned Figures 6-20 The corresponding method output is as follows, which will not be elaborated here.

[0362] Please see Figure 25 The following is a schematic diagram of another neural network distillation device provided in this application.

[0363] The neural network distillation apparatus may include a processor 2501 and a memory 2502. The processor 2501 and the memory 2502 are interconnected via circuits. The memory 2502 stores program instructions and data.

[0364] The aforementioned are stored in memory 2502 Figures 6 to 20 The steps in the code include the corresponding program instructions and data.

[0365] Processor 2501 is used to perform the aforementioned Figures 6 to 20 The method steps performed by the neural network distillation apparatus shown in any of the embodiments.

[0366] Optionally, the neural network distillation apparatus may also include a transceiver 2503 for receiving or sending data.

[0367] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 6 to 20 The steps in the method described in the illustrated embodiment.

[0368] Alternatively, the aforementioned Figure 25 The neural network distillation device shown is a chip.

[0369] This application also provides a target detection device, the structure of which is the same as described above. Figure 25 The structure is similar to that described above, and will be used as an example below. Figure 25 The structural schematic diagram of the target detection device provided in this application is illustrated by example.

[0370] The target detection device may include a processor 2501 and a memory 2502. The processor 2501 and the memory 2502 are interconnected via a circuit. The memory 2502 stores program instructions and data.

[0371] The aforementioned are stored in memory 2502 Figures 6 to 20 The steps in the code include the corresponding program instructions and data.

[0372] Processor 2501 is used to perform the aforementioned Figure 21 The method steps performed by the target detection device shown in any of the embodiments.

[0373] Optionally, the target detection device may also include a transceiver 2503 for receiving or sending data.

[0374] This application also provides a computer-readable storage medium storing a program that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 6 to 20 The steps in the method described in the illustrated embodiment.

[0375] Alternatively, the aforementioned Figure 25 The target detection device shown is a chip.

[0376] This application also provides a neural network distillation apparatus, which can also be referred to as a digital processing chip or chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figures 6 to 20 The method steps performed by the neural network distillation apparatus shown in any of the embodiments.

[0377] This application embodiment also provides a target detection device, which can also be referred to as a digital processing chip or a chip. The chip includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and the program instructions are executed by the processing unit. The processing unit is used to perform the aforementioned... Figure 21 The method steps performed by the target detection device shown in any of the embodiments.

[0378] This application also provides a digital processing chip. This digital processing chip integrates circuitry for implementing the processor 2501 described above, or the functions of processor 2501, and one or more interfaces. When the digital processing chip integrates a memory, it can complete the method steps of any one or more of the foregoing embodiments. When the digital processing chip does not integrate a memory, it can be connected to an external memory via a communication interface. The digital processing chip implements the actions performed by the neural network distillation device in the foregoing embodiments based on the program code stored in the external memory.

[0379] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned actions. Figures 6 to 20 The steps performed by the neural network distillation apparatus in the method described in the illustrated embodiment.

[0380] The neural network distillation device provided in this application embodiment can be a chip, which includes a processing unit and a communication unit. The processing unit can be, for example, a processor, and the communication unit can be, for example, an input / output interface, pins, or circuits. The processing unit can execute computer execution instructions stored in a storage unit to cause the chip within the server to perform the aforementioned operations. Figures 6 to 20 The neural network construction method described in the illustrated embodiment. Optionally, the storage unit is a storage unit within the chip, such as a register or cache. Alternatively, the storage unit can be a storage unit located outside the chip within the wireless access device, such as a read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions, such as random access memory (RAM).

[0381] Specifically, the aforementioned processing unit or processor can be a central processing unit (CPU), a neural-network processing unit (NPU), a graphics processing unit (GPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0382] For example, please refer to Figure 26 , Figure 26 This is a schematic diagram of a chip provided in an embodiment of this application. The chip can be represented as a neural network processor (NPU) 260. The NPU 260 is mounted as a coprocessor on the host CPU, and tasks are assigned by the host CPU. The core of the NPU is the arithmetic circuit 2603, which is controlled by a controller 2604 to retrieve matrix data from the memory and perform multiplication operations.

[0383] In some implementations, the arithmetic circuit 2603 internally includes multiple process engines (PEs). In some implementations, the arithmetic circuit 2603 is a two-dimensional pulsating array. The arithmetic circuit 2603 can also be a one-dimensional pulsating array or other electronic circuitry capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit 2603 is a general-purpose matrix processor.

[0384] For example, suppose we have an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit retrieves the corresponding data of matrix B from the weight memory 2602 and caches it in each PE of the arithmetic circuit. The arithmetic circuit retrieves the data of matrix A from the input memory 2601 and performs matrix operations with matrix B. The partial result or the final result of the obtained matrix is ​​stored in the accumulator 2608.

[0385] Unified memory 2606 is used to store input and output data. Weight data is directly transferred to weight memory 2602 via direct memory access controller (DMAC) 2605. Input data is also transferred to unified memory 2606 via DMAC.

[0386] The bus interface unit (BIU) 2610 is used for interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2609.

[0387] The bus interface unit 2610 (BIU) is used by the instruction fetch memory 2609 to fetch instructions from external memory, and also by the memory access controller 2605 to fetch the original data of the input matrix A or the weight matrix B from external memory.

[0388] The DMAC is mainly used to move input data from external memory DDR to unified memory 2606, or to weight data to weight memory 2602, or to input data to input memory 2601.

[0389] The vector computation unit 2607 includes multiple processing units that further process the output of the computation circuits as needed, such as vector multiplication, vector addition, exponential operations, logarithmic operations, size comparisons, etc. It is primarily used for computations in non-convolutional / fully connected layers of neural networks, such as batch normalization, pixel-level summation, and upsampling of feature planes.

[0390] In some implementations, vector computation unit 2607 can store the processed output vector in unified memory 2606. For example, vector computation unit 2607 can apply linear and / or nonlinear functions to the output of computation circuit 2603, such as performing linear interpolation on feature planes extracted from convolutional layers, or accumulating a vector of values ​​to generate activation values. In some implementations, vector computation unit 2607 generates normalized values, pixel-level summed values, or both. In some implementations, the processed output vector can be used as activation input to computation circuit 2603, for example, for use in subsequent layers of the neural network.

[0391] The instruction fetch buffer 2609 connected to the controller 2604 is used to store the instructions used by the controller 2604;

[0392] The unified memory 2606, input memory 2601, weighted memory 2602, and instruction fetch memory 2609 are all on-chip memories. External memory is proprietary to this NPU hardware architecture.

[0393] The operations of each layer in the recurrent neural network can be performed by the arithmetic circuit 2603 or the vector calculation unit 2607.

[0394] The processor mentioned above can be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more processors used to control the above. Figures 6-22 The method of program execution of integrated circuits.

[0395] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.

[0396] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0397] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.

[0398] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or data center that integrates one or more available media. The available medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0399] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in a sequence other than that illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0400] Finally, it should be noted that the above are only specific embodiments of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A neural network distillation method, characterized in that, include: Obtain the teacher model and the first student model; The first student model is evolved to obtain an evolved first student model, wherein the evolution includes: transforming the structure of the first student model according to the evolution action to obtain the evolved first student model, wherein the evolution action includes at least one of adding a layer, layer rearrangement, or pruning, wherein adding a layer indicates adding at least one network layer in the first student model, layer rearrangement indicates rearranging at least one network layer in the first student model, and pruning indicates reducing the structure of the first student model; Distillation is performed on the evolved first student model using the teacher model to obtain an updated first student model; The distillation of the evolved first student model using the teacher model includes: The evolved first student model is distilled using the teacher model through the training set to obtain the updated first student model. The training set includes at least one image sample. The updated first student model is used to perform at least one of the following tasks: object detection, classification, or segmentation. The acquisition of the teacher model and the first student model includes: The teacher model and the first student model are obtained from the model pair list, wherein the model pair list includes at least one model pair, and each model pair includes the structure of a teacher model and the structure of a student model.

2. The method according to claim 1, characterized in that, If the updated first student model does not meet the preset conditions, a second student model is obtained, the second student model is evolved, and the evolved second student model is distilled using the teacher model or a new teacher model to obtain an updated second student model.

3. The method according to claim 1, characterized in that, The method further includes: If the updated first student model does not meet the preset conditions, then the teacher model and the updated first student model are added as a model pair to the model pair list.

4. The method according to any one of claims 1 to 3, characterized in that, The step of obtaining the teacher model and the first student model from the model list includes: Model pairs whose evaluation results are higher than the preset results are selected from the list of model pairs; The teacher model and the first student model are obtained based on the structure of the model pair.

5. The method according to any one of claims 1 to 3, characterized in that, The method further includes: Construct a hypernetwork, which includes multiple subnetworks; The supernetwork is trained at least once using the training set to obtain the trained subnetworks, and the teacher model is collected from the trained subnetworks.

6. The method according to claim 5, characterized in that, Any one of the at least one training sessions includes: At least one subnetwork is acquired from the supernetwork; The at least one sub-network is trained to obtain the trained multiple sub-networks.

7. The method according to claim 6, characterized in that, The step of collecting at least one sub-network from the supernetwork includes: Structural parameters are acquired from a structural sampling space, which includes one or more of the following: depth range, width range, or kernel size range. The width range is the range of the number of input and output channels of the basic units included in the sampled sub-network. The depth range is the range of the number of network layers in the sampled sub-network. The kernel size range includes the range of the size of the kernels in the sampled sub-network. The at least one sub-network is obtained from the supernetwork based on the structural parameters.

8. The method according to claim 7, characterized in that, In each of the at least one training sessions, no more than two types of structural parameters are collected from the structural sampling space. The at least one sub-network includes a first sub-network and a second sub-network, the structure of the first sub-network is larger than the structure of the second sub-network, and the first sub-network is trained before the second sub-network.

9. The method according to claim 2 or 3, characterized in that, The preset conditions include one or more of the following: output accuracy is greater than a first threshold, or average accuracy is greater than a second threshold, or loss value is not greater than a third threshold, or inference time is not greater than a fourth threshold, or floating-point operations (FLOPs) are not greater than a fifth threshold, or comprehensive score is greater than a sixth threshold. The average accuracy is the average of multiple accuracies obtained by evaluating the updated first student model multiple times. The inference time is the time it takes for the neural network to obtain the output result based on the input. The comprehensive score is obtained by weighting at least two of the output accuracy, average accuracy, loss value, inference time, or FLOPs.

10. A target detection method, characterized in that, include: Obtain the input image; The input image is used as the input to the target model, and the information of the objects included in the input image is output. The target model is an updated student model obtained by knowledge distillation of the student model using the teacher model. The teacher model and the student model are obtained according to the model pair list, which includes at least one model pair. Each model pair includes the structure of a teacher model and the structure of a student model. The knowledge distillation of the student model using the teacher model includes: evolving the student model to obtain an evolved student model; and performing knowledge distillation of the evolved student model using the teacher model to obtain an updated student model. The evolution includes: transforming the structure of the student model according to an evolution action to obtain the evolved student model. The evolution action includes at least one of adding a layer, layer rearrangement, or pruning. Adding a layer indicates adding at least one network layer to the student model. Layer rearrangement indicates rearranging at least one network layer in the student model. Pruning indicates reducing the structure of the student model.

11. The method according to claim 10, characterized in that, The information of the objects included in the input image includes: the category of the object and / or the location of the object in the input image.

12. A neural network distillation apparatus, characterized in that, include: The acquisition module is used to acquire the teacher model and the first student model; An evolution module is used to perform at least one evolution on the first student model to obtain an evolved first student model. Each of the at least one evolutions includes: transforming the structure of the first student model according to an evolution action to obtain the evolved first student model. The evolution action includes at least one of adding a layer, layer rearrangement, or pruning. Adding a layer indicates adding at least one network layer to the first student model. Layer rearrangement indicates rearranging at least one network layer in the first student model. Pruning indicates reducing the structure of the first student model. A distillation module is used to distill the evolved first student model using the teacher model to obtain an updated first student model. The distillation module is specifically used to distill the evolved first student model using the training set through the teacher model to obtain the updated first student model, which is used to perform at least one of the following tasks: object detection, classification, or segmentation. The acquisition module is specifically used to obtain the teacher model and the first student model from a list of model pairs. The list of model pairs includes at least one pair of model pairs, and each pair of model pairs includes the structure of a teacher model and the structure of a student model.

13. The apparatus according to claim 12, characterized in that, If the updated first student model does not meet the preset conditions, the acquisition module acquires the second student model, the evolution module evolves the second student model, and the distillation module uses the teacher model or a new teacher model to distill the evolved second student model to obtain the updated second student model.

14. The apparatus according to claim 12, characterized in that, The acquisition module is further configured to add the teacher model and the updated first student model as a model pair to the model pair list if the updated first student model does not meet the preset conditions.

15. The apparatus according to any one of claims 12 to 14, characterized in that, The acquisition module is specifically used for: Model pairs whose evaluation results are higher than the preset results are selected from the list of model pairs; The teacher model and the first student model are obtained based on the structure of the model pair.

16. The apparatus according to any one of claims 12 to 14, characterized in that, The device further includes: A building module is used to build a hypernetwork, which includes multiple subnetworks; The training module is used to train the supernetwork at least once using the training set to obtain multiple subnetworks after training, and the teacher model is collected from the multiple subnetworks after training.

17. The apparatus according to claim 16, characterized in that, Any one of the at least one training sessions includes: At least one subnetwork is acquired from the supernetwork; The at least one sub-network is trained to obtain the trained supernetwork.

18. The apparatus according to claim 17, characterized in that, The training module is specifically used for: Structural parameters are acquired from a structural sampling space, which includes one or more of the following: depth range, width range, or kernel size range. The width range is the range of the number of input and output channels of the basic units included in the sampled sub-network. The depth range is the range of the number of network layers in the sampled sub-network. The kernel size range includes the range of the size of the kernels in the sampled sub-network. The at least one sub-network is obtained from the supernetwork based on the structural parameters.

19. The apparatus according to claim 18, characterized in that, In each of the at least one training sessions, no more than two types of structural parameters are collected from the structural sampling space. The at least one sub-network includes a first sub-network and a second sub-network, the structure of the first sub-network is larger than the structure of the second sub-network, and the first sub-network is trained before the second sub-network.

20. The apparatus according to claim 13 or 14, characterized in that, The preset conditions include one or more of the following: output accuracy is greater than a first threshold, or average accuracy is greater than a second threshold, or loss value is not greater than a third threshold, or inference time is not greater than a fourth threshold, or floating-point operations (FLOPs) are not greater than a fifth threshold, or comprehensive score is greater than a sixth threshold. The average accuracy is the average of multiple accuracies obtained by evaluating the updated first student model multiple times. The inference time is the time it takes for the neural network to obtain the output result based on the input. The comprehensive score is obtained by weighting at least two of the output accuracy, average accuracy, loss value, inference time, or FLOPs.

21. A target detection device, characterized in that, include: The transceiver module is used to acquire the input image; The output module is used to take the input image as input to the target model and output the information of the objects included in the input image. The target model is an updated student model obtained by knowledge distillation of the student model using the teacher model. The teacher model and the student model are obtained according to a model pair list. The model pair list includes at least one model pair, and each model pair includes the structure of a teacher model and the structure of a student model. The updated student model obtained by knowledge distillation of the student model using the teacher model includes: evolving the student model to obtain an evolved student model, and performing knowledge distillation on the evolved student model using the teacher model to obtain the updated student model. The evolution includes: transforming the structure of the student model according to the evolution action to obtain the evolved student model. The evolution action includes at least one of adding a layer, layer rearrangement, or pruning. Adding a layer indicates adding at least one network layer to the student model. Layer rearrangement indicates rearranging at least one network layer in the student model. Pruning indicates reducing the structure of the student model.

22. The apparatus according to claim 21, characterized in that, The information of the objects included in the input image includes: the category of the object and / or the location of the object in the input image.

23. A neural network distillation apparatus, characterized in that, The method includes a processor coupled to a memory storing a program, wherein the program instructions stored in the memory are executed by the processor to implement the method of any one of claims 1 to 9.

24. A target detection device, characterized in that, The system includes a processor coupled to a memory storing a program that, when executed by the processor, implements the method of claim 10 or 11.

25. A computer-readable storage medium comprising a program, which, when executed by a processing unit, performs the method as claimed in any one of claims 1 to 9 or 10 to 11.

26. An apparatus, characterized in that, It includes a processing unit and a communication interface. The processing unit obtains program instructions through the communication interface, and when the program instructions are executed by the processing unit, it implements the method of any one of claims 1 to 9 or 10 to 11.

Images