A neural network search method and apparatus, electronic device

Abstract

This application discloses a neural network search method, apparatus, and electronic device, relating to the field of artificial intelligence. The method includes: obtaining a first sub-network from a supernet of a first neural network architecture search; constructing a second neural network architecture search supernet based on the first sub-network; using the first neural network architecture search supernet as a teacher network in knowledge distillation to train the second neural network architecture search supernet; and searching for a target sub-network from the trained second neural network architecture search supernet using joint performance constraints of computational power and accuracy. This application enables the second neural network architecture search supernet to have the same performance as the first sub-network, allows the trained second neural network architecture search supernet to achieve better network performance, and enables the target sub-network to meet computational power and accuracy requirements, realizing the possibility of searching for a sub-network with both excellent computational power and accuracy.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, specifically to a neural network search method and apparatus, and electronic equipment. Background Technology

[0002] Neural Architecture Search (NAS) is one of the hottest research topics in the field of artificial intelligence in recent years.

[0003] The core elements of the NAS algorithm include the search space, search strategy, and performance estimation strategy. The search space defines the set of searchable neural network structures, i.e., the solution space. The search strategy defines how to find the optimal neural network structure within the search space. The performance estimation strategy defines how to evaluate the network performance of the searched neural network structure, and the network performance can be measured by metrics such as accuracy and computational power (or computing power).

[0004] The principle of the NAS algorithm can be summarized as follows: Define a search space; in each iteration of the search process, a neural network structure called a "sub-network" is obtained from the search space through a search strategy; train the sub-network on the training sample set, and evaluate the performance of the sub-network on the validation set through a performance evaluation strategy, so as to gradually optimize the network structure until the sub-network with the best performance is found.

[0005] However, with the increasing development of technology in the field of artificial intelligence, further research is needed on how to perform neural network search through NAS. Summary of the Invention

[0006] This application provides a neural network search method, apparatus, and electronic device to address the problem of how to perform neural network search via NAS.

[0007] Firstly, this application provides a neural network search method, including...

[0008] Obtain the first subnetwork in the supernet searched by the first neural network architecture;

[0009] The supernet is searched by constructing a second neural network architecture based on the first sub-network;

[0010] The supernet searched by the first neural network architecture is used as the teacher network in knowledge distillation to train the supernet searched by the second neural network architecture.

[0011] The target subnetwork is searched from the supernetwork of the trained second neural network architecture using performance constraints of joint computing power and accuracy.

[0012] As can be seen, this application constructs a second neural network architecture search supernet by using a first sub-network in a supernet searched by a first neural network architecture, enabling the second neural network architecture search supernet to have the same performance as the first sub-network. This suggests that if the first sub-network has the best network performance, the second neural network architecture search supernet may also have the best network performance. During the training of the second neural network architecture search supernet, knowledge distillation is used to treat the first neural network architecture search supernet as a teacher network to train the second neural network architecture search supernet, resulting in better network performance after training. From the trained second neural network architecture search supernet, a target subnet is searched using joint performance constraints of computational power and accuracy, ensuring that the target subnet meets the computational power and accuracy requirements, thus realizing the possibility of finding a subnet with both excellent computational power and accuracy.

[0013] Secondly, this application provides a neural network search device, comprising:

[0014] The acquisition unit is used to acquire the first subnetwork in the supernet searched by the first neural network architecture.

[0015] A construction unit is used to construct a supernet for the second neural network architecture search based on the first sub-network;

[0016] The training unit is used to train the supernet searched by the first neural network architecture as a teacher network in knowledge distillation to train the supernet searched by the second neural network architecture.

[0017] The search unit is used to search for the target subnetwork from the supernet of the trained second neural network architecture with performance constraints of joint computing power and accuracy.

[0018] Thirdly, an electronic device according to this application includes a processor, a memory, and a computer program or instructions stored in the memory, wherein the processor executes the computer program or instructions to implement the steps of the method designed in the first aspect above.

[0019] Fourthly, this application provides a computer-readable storage medium, wherein a computer program or instructions are stored on the computer-readable storage medium, and when executed by a processor, the computer program or instructions implement the steps of the method designed in the first aspect above.

[0020] Fifthly, this application provides a computer program product comprising a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps of the method designed in the first aspect above.

[0021] The beneficial effects of the technical solutions in the second to fifth aspects can be found in the technical effects of the technical solution in the first aspect, and will not be repeated here. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below.

[0023] Figure 1 This is an overall flowchart of a neural network search method according to an embodiment of this application;

[0024] Figure 2 This is an overall flowchart of a supernet for training a second neural network architecture search according to an embodiment of this application;

[0025] Figure 3 This is an overall flowchart of a search target sub-network according to an embodiment of this application;

[0026] Figure 4 This is a flowchart illustrating a neural network search method according to an embodiment of this application;

[0027] Figure 5 This is a functional unit block diagram of a neural network search device according to an embodiment of this application;

[0028] Figure 6 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0029] To help those skilled in the art better understand the technical solutions of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the description of the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0030] It should be understood that the terms "first," "second," etc., used in the embodiments of this application are used to distinguish different objects, rather than to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, software, product, or device that includes a series of steps or units is not limited to the listed steps or units, but also includes steps or units not listed, or other steps or units inherent to these processes, methods, products, or devices.

[0031] The term "embodiment" as used in the embodiments of this application means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0032] In the embodiments of this application, "and / or" describes the relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent the following three situations: A exists alone; A and B exist simultaneously; B exists alone. Among them, A and B can be singular or plural.

[0033] In this embodiment, the symbol " / " can indicate that the preceding and following objects are in an "or" relationship. Alternatively, the symbol " / " can also represent a division sign, i.e., performing a division operation. For example, A / B can mean A divided by B.

[0034] In the embodiments of this application, "at least one item" or its similar expression refers to any combination of these items, including any combination of a single item or a plurality of items. "One or more" means one or more, while "multiple" means two or more. For example, "at least one item" of a, b, or c can represent the following seven cases: a, b, c; a and b; a and c; b and c; a, b, and c. Each of a, b, and c can be an element or a set containing one or more elements.

[0035] In the embodiments of this application, "equal to" can be used with "greater than" and is applicable to technical solutions used when "greater than" is used; it can also be used with "less than" and is applicable to technical solutions used when "less than" is used. When "equal to" is used with "greater than", it is not used with "less than"; when "equal to" is used with "less than", it is not used with "greater than".

[0036] With the continuous development of artificial intelligence technology, further research is needed on how to perform neural network search through Neural Architecture Search (NAS).

[0037] The technical solutions and related concepts involved in the embodiments of this application will be described in detail below.

[0038] I. Electronic Equipment

[0039] The technical solutions of this application embodiment can be executed by an electronic device. This electronic device can be a handheld device, in-vehicle device, wearable device, augmented reality (AR) device, virtual reality (VR) device, projection device, projector, or other device connected to a wireless modem. It can also be user equipment (UE), terminal device, terminal, mobile terminal, smartphone, smart screen, smart TV, smartwatch, laptop, smart speaker, camera, game controller, microphone, station (STA), access point (AP), mobile station (MS), personal digital assistant (PDA), personal computer (PC), or relay device, etc.

[0040] In this embodiment of the application, the electronic device may include a processor.

[0041] In some possible implementations, the processor can be used to run or load an operating system, which can be any one or more computer operating systems that perform business processing through processes. Examples include Linux, Unix, Android, iOS, Windows, Zephyr, Real-Time Operating System (RTOS), DOS, Mac, ThreadX, embedded operating systems, and Nucleus Plus.

[0042] In some possible implementations, the processor can be viewed as a complete system-on-chip (SOC).

[0043] In some possible implementations, the processor may include one or more processing units. For example, a processing unit may include at least one of the following: a central processing unit (CPU), an application processor (AP), a microcontroller unit (MCU), a single-chip microcomputer (SCM), a microcontroller, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a digital signal processor (DSP), a field-programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a baseband processor, and a neural network processing unit (NPU). The different processing units may be separate or integrated together.

[0044] In some possible implementations, a processing unit can be a single core or multiple cores.

[0045] In some possible implementations, a processing unit can run or load a multi-core subsystem. This multi-core subsystem can be an operating system with multi-core processing capabilities.

[0046] In some possible implementations, the processor may also include memory for storing instructions and data.

[0047] For example, a processor can call programs stored in memory to run an operating system.

[0048] For example, the processor's memory can store or cache instructions that the processor has just used or that are used repeatedly. If the processor needs to use the instruction or data again, it can retrieve it directly from that memory, thereby avoiding repeated accesses, reducing processor wait time, and improving system efficiency.

[0049] For example, the memory in a processor can also store or cache data, and synchronize or transfer this data to other processors for execution. This memory in the processor can be a cache memory.

[0050] In some possible implementations, the processor may include one or more communication interfaces. These communication interfaces may include at least one of the following: Serial Peripheral Interface (SPI), Inter-Integrated Circuit (I2C) interface, Inter-Integrated Circuit Sound (I2S) interface, Pulse Code Modulation (PCM) interface, Universal Asynchronous Receiver / Transmitter (UART) interface, Mobile Industry Processor Interface (MIPI), General-Purpose Input / Output (GPIO) interface, Subscriber Identity Module (SIM) interface, and Universal Serial Bus (USB) interface.

[0051] In the embodiments of this application, the hardware and software structure of the electronic device may include a hardware layer, an operating system layer running on the hardware layer, and an application layer running on the operating system layer.

[0052] In some possible implementations, this hardware layer may include hardware such as a processor, a memory management unit (MMU), and memory (also known as storage).

[0053] In some possible implementations, the memory can be used to store software programs / computer programs / computer instructions / data, etc., and may include a program storage area and a data storage area. The program storage area can be used to store software programs / computer programs / computer instructions, etc., required by the operating system or at least one function, and the software programs / computer programs / computer instructions required by the at least one function can be used to execute the technical solutions of the embodiments of this application; the data storage area can be used to store data, etc., involved in the embodiments of this application.

[0054] It should be noted that the embodiments of this application do not impose any particular limitation on the specific structure of the subject implementing this solution, as long as it can be processed by running a computer program or instructions that record the methods provided in the embodiments of this application, or by processing according to the methods provided in the embodiments of this application. For example, the subject implementing the technical solution of the embodiments of this application can be an electronic device, or a processor / device / module / unit in an electronic device that can call computer programs or instructions to execute, etc., and there are no specific limitations in this regard.

[0055] II. NAS

[0056] The core elements of NAS algorithms include search space, search strategy, and performance evaluation strategy.

[0057] 1. Search Space

[0058] The search space defines the types and set of neural network structures that the NAS algorithm can search, i.e., the solution space, and also defines how the neural network structure should be described.

[0059] The computations performed by neural networks can be abstracted as a directed acyclic graph (DAG) without isolated nodes. In this DAG, the nodes represent the layers of the neural network or the feature maps in a convolutional network; the edges of the DAG represent the flow of data or operations (OPs), which can also be called operators, such as convolution and pooling.

[0060] Each node in a Directed Acyclic Graph (DAG) receives data from its predecessor (with incoming edges), performs computation, and then outputs the data to subsequent nodes (with outgoing edges). Theoretically, any DAG without isolated nodes is a valid neural network structure. Depending on the scale, the structural definition of a neural network includes the following levels of information:

[0061] (1) Network topology

[0062] The topology of a network describes the number of layers in a neural network and the connections between them. The simplest neural network is a linear chain structure, which corresponds to at most one predecessor node and one successor node in its graph, similar to a linked list in data structures. When describing the topology of a network, the predecessor node is typically used for definition; that is, the predecessor node of each node is defined. Once this information is determined, the network topology is determined.

[0063] (2) Type of each layer

[0064] In a neural network architecture, the first layer is the input layer, the last layer is the output layer, and the types of the intermediate layers are optional; they represent various operations (operators), i.e., the layer types. Typical examples include fully connected layers, convolutional layers, deconvolutional layers, separable convolutional layers, dilated convolutional layers, pooling layers, and activation functions.

[0065] (3) Hyperparameters within each layer

[0066] The hyperparameters of a convolutional layer include the kernel size, the number of kernels, the number of channels in the kernel, height, width, horizontal stride, and vertical stride.

[0067] The hyperparameters of the pooling layer include the pooling kernel height, pooling kernel width, horizontal step size, and vertical step size.

[0068] The hyperparameters of a fully connected layer include the number of neurons.

[0069] The hyperparameters of the activation function layer include the type of the activation function and the parameters of the activation function.

[0070] To improve search efficiency, the search space is sometimes limited or simplified. For example, the network can be divided into basic units, and more complex networks can be formed by stacking these cells. A cell consists of multiple nodes and edges that appear repeatedly throughout the network but have different weight parameters.

[0071] 2. Search Strategy

[0072] Search strategies define how to find the optimal neural network structure in the search space. For example, search strategies include reinforcement learning-based search strategies, genetic algorithm-based search strategies, gradient descent-based search strategies, and so on.

[0073] 3. Performance Evaluation Strategy

[0074] The performance evaluation strategy defines how to evaluate the performance of the searched neural network architecture, and the performance of the neural network architecture can be measured by metrics such as accuracy and computing power.

[0075] III. Search for Other Neural Network Architectures

[0076] 1. Differentiable Architecture Search (DARTS)

[0077] Unlike most neural network architecture searches that employ reinforcement learning or genetic algorithms, DARTS uses a gradient descent-based search strategy. Instead of searching a discrete set of neural network architectures, DARTS relaxes the search space to be continuous, allowing for optimization of the neural network architecture's performance on the validation set using gradient descent.

[0078] DARTS relaxes the discrete search space to continuous, allowing the search strategy to optimize the performance of the neural network structure on the validation set using gradient descent, ultimately solving the neural network search problem.

[0079] In DARTS, the search space is constructed using a stacked cell approach. First, each node in a cell represents a feature map, and all nodes are connected by at least one edge, with each edge representing an operator (operation). The number of nodes in each cell and the types of operators in the search space must also be determined. Second, based on the operators given in the search space, each operator is assigned an initial value for its corresponding weight parameter. The connections between nodes are determined, and a directed acyclic graph (DAG) is constructed to represent the cell structure. Third, the weight parameters for each operator are updated using a gradient descent search strategy based on the architecture parameters. Once all weight parameters converge, the operator with the largest architecture parameter between each pair of nodes is selected as the final operator in the cell. This process yields the desired neural network structure.

[0080] 2. Train one network and specialize it for efficient deployment (Once-For-All)

[0081] Most NAS algorithms can only be trained and searched for a specific device or a platform with limited resources, and cannot be adapted to other devices or platforms with different resource constraints. In other words, for other devices or platforms with different resource constraints, the NAS needs to be retrained before searching, resulting in poor scalability and high computational cost. To address this, Once-For-All aims to decouple the training and search processes, allowing the training of a supernetwork that supports different architecture configurations. A subnetwork can then be obtained by searching from the trained Once-For-All supernetwork without retraining that subnetwork.

[0082] IV. Knowledge Distillation

[0083] Knowledge distillation is a common method in model compression. Unlike pruning and quantization in model compression, knowledge distillation involves constructing a lightweight, smaller network and training it using the supervision information from a larger, more powerful network to achieve better overall network performance. The larger network is called the teacher network, and the smaller network is called the student network. The supervision information output by the teacher network is called knowledge, and the process by which the student network learns and transfers this supervision information is called distillation.

[0084] V. A Neural Network Search Method

[0085] To achieve neural network search via NAS, this application proposes a neural network search method. A second neural network architecture search supernet is constructed using a first sub-network from a supernet searched by a first neural network architecture. This allows the second neural network architecture search supernet to have the same performance as the first sub-network, potentially ensuring optimal performance for the second neural network architecture search supernet if the first sub-network achieves the best performance. During the training of the second neural network architecture search supernet, knowledge distillation is used to employ the first neural network architecture search supernet as a teacher network, resulting in improved network performance. Finally, a target sub-network is searched from the trained second neural network architecture search supernet using joint performance constraints of computational power and accuracy. This ensures the target sub-network meets both computational power and accuracy requirements, potentially yielding a sub-network with excellent computational power and accuracy.

[0086] 1. Related Concepts

[0087] ① First Neural Network Architecture

[0088] In the embodiments of this application, the first neural network architecture can be a NAS based on a gradient descent search strategy, a NAS with a continuously relaxed search space, or a NAS with a continuously relaxed search space and a gradient descent search strategy.

[0089] For example, the first neural network architecture may include differentiable architecture search, such as DARTS.

[0090] ② Second Neural Network Architecture

[0091] In the embodiments of this application, the second neural network architecture can be a NAS with decoupled training and search processes.

[0092] For example, the second neural network architecture could include Once-For-All.

[0093] 2. Implementation Principle

[0094] like Figure 1 As shown, Figure 1 This is an overall flowchart of a neural network search method according to an embodiment of this application. The specific steps are as follows:

[0095] Step 1: Define the search space for the first neural network architecture

[0096] It should be noted that the search space of the first neural network architecture can include a list of operators. The operator types can include convolution, pooling, identity, etc.

[0097] Step 2: Construct the supernet of the first neural network architecture

[0098] It should be noted that the supernet of the first neural network architecture can be viewed as a stack of cells. Here, a cell represents a directed acyclic graph (DAG). Each node in the DAG represents a feature map, and all nodes are connected by at least one edge, with each edge representing an operator (operation). Each edge corresponds to a weight parameter (such as the weights of a convolutional kernel).

[0099] Step 3: Obtain training data

[0100] Step 4: Train the supernet of the first neural network architecture

[0101] It should be noted that, in this embodiment of the application, the training data obtained in step 3 can be used to train the supernet of the first neural network architecture.

[0102] Step 5: Search to obtain the first subnetwork

[0103] It should be noted that, in this embodiment of the application, the first subnetwork can be obtained by searching from the supernetwork of the trained first neural network architecture. The search strategy employed is a gradient descent-based search strategy.

[0104] Furthermore, the first subnetwork can be a subnetwork whose accuracy meets a preset requirement, where the architecture parameters corresponding to the edges (operators / operations) connecting each pair of nodes in this subnetwork are maximized. In other words, the first subnetwork is the subnetwork with the best accuracy in the supernet of the first neural network architecture.

[0105] To ensure the performance of the first sub-network, embodiments of this application may retrain the first sub-network. For example, the first sub-network may be retrained to ensure that the accuracy meets the requirements.

[0106] Step 6: Dynamically adjust the first subnetwork

[0107] It should be noted that the first sub-network has a fixed network width (i.e., the number of channels per layer in the first sub-network), network depth (i.e., the number of layers in the first sub-network), convolutional kernel size (i.e., the size of the convolutional kernels in the first sub-network), number of convolutional kernels (i.e., the number of convolutional kernels in the first sub-network), and feature map resolution (i.e., the feature map resolution of each layer in the first sub-network). Therefore, embodiments of this application can dynamically adjust at least one of the network width, network depth, convolutional kernel size, number of convolutional kernels, and feature map resolution in the first sub-network to obtain a baseline model.

[0108] In addition, dynamic adjustments can include increasing, decreasing, or keeping the number of convolutional kernels constant. For example, increasing the network width, decreasing the network depth, increasing the kernel size, keeping the number of kernels constant, and increasing the feature map resolution.

[0109] For example, increase the kernel size from 3×3 to 5×5.

[0110] Step 7: Construct the supernet for the second neural network architecture search.

[0111] It should be noted that, in the embodiments of this application, the benchmark model can be used as the supernet for the second neural network architecture search, thereby realizing the construction of the supernet for the second neural network architecture search based on the first sub-network, so that the supernet for the second neural network architecture search can have the same performance as the first sub-network. In this way, if the first sub-network has the best accuracy, the supernet for the second neural network architecture search can also have the best accuracy.

[0112] Step 8: Train the supernet for the second neural network architecture search.

[0113] It should be noted that, in this embodiment of the application, knowledge distillation can be used to train the supernet of the second neural network architecture search as a teacher network, so that the trained supernet of the second neural network architecture search can achieve better network performance.

[0114] In specific implementation, such as Figure 2 As shown, the steps for training the supernet for the second neural network architecture search are as follows:

[0115] Step 801: Constructing a Teacher Network for Knowledge Distillation

[0116] It should be noted that, in the embodiments of this application, the supernet searched by the first neural network architecture can be used as the teacher network in knowledge distillation, thereby realizing the construction of the teacher network.

[0117] Step 802: Extract a subnetwork from the supernet searched by the second neural network architecture.

[0118] It should be noted that, in the training process of the supernet for the second neural network architecture search in this embodiment, a subnetwork needs to be sequentially collected from the network of the second neural network architecture search, and this subnetwork is used as a student network for training, until a certain number of subnetworks in the supernet for the second neural network architecture search are trained to complete the training of the supernet for the second neural network architecture search. Since this embodiment does not need to train all subnetworks in the supernet for the second neural network architecture search, but only a certain number of subnetworks, training time is saved and training efficiency is improved.

[0119] For ease of distinction and description, in the embodiments of this application, a subnetwork collected from the supernet searched by the second neural network architecture can be referred to as the "second subnetwork".

[0120] For example, the second subnetwork is the i-th subnetwork in the supernet searched by the second neural network architecture, where i represents the subnetwork number of the second subnetwork (or student network) in the supernet searched by the second neural network architecture.

[0121] Furthermore, in this embodiment of the application, a sub-network can be acquired through random sampling. In other words, the second sub-network can be acquired through random sampling.

[0122] Step 803: Constructing the student network in knowledge distillation

[0123] It should be noted that, in the embodiments of this application, the second sub-network can be used as the student network in knowledge distillation, thereby realizing the construction of students.

[0124] In other words, in this embodiment of the application, a subnetwork can be selected sequentially from the supernet searched by the second neural network architecture as the student network.

[0125] Step 804: Input the first input data into the teacher network and student network.

[0126] It should be noted that the first input data can be a type of data, such as an image, data, feature points, etc. In this embodiment of the application, the first input data can be input to the teacher network and the student network to obtain their respective output results.

[0127] Step 805: Input the first input data into the Ground Truth and student network.

[0128] It should be noted that, in this embodiment of the application, first input data can be input to the teacher network and the student network to obtain their respective output results.

[0129] Step 806: Calculate the value of the loss function.

[0130] It should be noted that, during the process of inputting the first input data into the teacher network and the student network, this embodiment of the application needs to calculate the value of the loss function based on the output results between layers. For ease of distinction and description, this embodiment of the application refers to the value of the loss function between the output results of the j-th layer in the student network and the j-th layer in the teacher network for the first input data as the "first loss value".

[0131] For example, calculate the loss function R(i,j) between the output of the j-th layer in the student network and the output of the j-th layer in the teacher network, where j represents the layer number.

[0132] In the process of inputting the first input data into the ground reality network and the student network, this embodiment of the application needs to calculate the value of the loss function according to the output results between layers. For ease of distinction and description, this embodiment of the application refers to the value of the loss function between the output results of the j-th layer in the student network and the j-th layer in the ground reality network for the first input data as the "second loss value".

[0133] For example, the loss function L(i,j) between the output of the j-th layer in the student network and the output of the j-th layer in the ground reality is calculated sequentially. By accumulating the loss function values ​​of each layer, the sum of these values ​​is obtained. j L(i,j).

[0134] In addition, since knowledge distillation is used, this embodiment of the application also needs to calculate the regularization of the loss function of knowledge distillation between the student network and the teacher network, and calculate the product between the first loss value and the regularization term coefficient to obtain the value of the regularized loss function.

[0135] For the j-th layer, the regularization term coefficients can be determined by the first loss value, the second loss value, and the weight parameters (such as the weights of convolutional layers) of the j-th layer in the second subnetwork (i.e., the student network). For ease of distinction and description, the value of the regularized loss function is called the "third loss value".

[0136] For example, the third loss value is a(i,j)*R(i,j), where a(i,j) represents the regularization coefficient. Here, a(i,j) is:

[0137] a(i,j)=cos(dR(i,j) / dw(i,j),dL(i,j) / dw(i,j));

[0138] Where dR(i,j) represents the gradient of R(i,j); dL(i,j) represents the gradient of L(i,j); w(i,j) represents the weight parameter of the j-th layer in the second sub-network (i.e., the student network); and cos() represents the cosine calculation.

[0139] It should be noted that the embodiments of this application can measure the vector difference between two gradient optimization vectors (i.e., dR(i,j) / dw(i,j) and dL(i,j) / dw(i,j)) by using cosine calculation, thereby improving the accuracy and precision of network optimization.

[0140] Finally, by iterating through each layer and summing the values ​​of the loss function for each layer, we obtain ∑ j a(i,j)*R(i,j).

[0141] Step 807: Train the second subnetwork

[0142] It should be noted that, based on the above steps, the optimization objective obtained in the training of the second sub-network in this embodiment of the application is:

[0143] Min∑ j L(i,j)+a(i,j)*R(i,j).

[0144] In other words, the second sub-network is trained with the goal of minimizing the value of the loss function (such as minimizing the sum of the second and third loss values).

[0145] Step 808: Repeat step 802.

[0146] It should be noted that, since this embodiment of the application requires training a certain number of subnetworks in the supernet of the second neural network architecture search to complete the training of the supernet of the second neural network architecture search, this embodiment of the application needs to repeatedly collect a subnetwork from the supernet of the second neural network architecture search, that is, repeatedly execute step 802, and then execute steps 803 to 807.

[0147] Step 809: Complete the training of the supernet for the second neural network architecture search.

[0148] Step 9: Search the target subnetwork

[0149] It should be noted that, in the embodiments of this application, the target subnetwork can be searched from the supernetwork searched by the trained second neural network architecture with performance constraints of joint computing power and accuracy.

[0150] Since the performance constraints are based on the combined computing power and accuracy, the target subnetwork can be a subnetwork that meets the preset requirements for both computing power and accuracy. In other words, the target subnetwork searched in the embodiments of this application can guarantee excellent computing power and accuracy.

[0151] In specific implementation, such as Figure 3 As shown, the steps for searching the target subnetwork are as follows:

[0152] Step 901: Collect training data

[0153] It should be noted that the embodiments of this application can collect training data to train the accuracy predictor. The accuracy predictor can be viewed as a predictive model used to calculate the accuracy of a sub-network, such as a multi-layer perceptron machine (MLP).

[0154] In some possible implementations, embodiments of this application can collect sub-networks from the supernet searched by the second neural network architecture, encode the sub-networks, and calculate the corresponding precision of the sub-networks to form data pairs in the training data. That is, the data pair can be represented as [the encoding of the sub-network, the precision of the sub-network]. Typically, embodiments of this application can collect 5000 data pairs to form training data to ensure training quality.

[0155] Step 902: Training the accuracy predictor

[0156] Step 903: Search for subnetworks that satisfy preset computing power constraints.

[0157] It should be noted that, in this embodiment of the application, subnetworks can be searched from the supernet searched by the trained second neural network architecture to meet preset computing power constraints, thereby obtaining at least one candidate subnetwork. Therefore, each candidate subnetwork meets the preset computing power requirements.

[0158] In some possible implementations, the preset computing power constraints may include the number of floating-point operations per second (FLOPs) or the number of parameters (#Params).

[0159] Step 904: Search in at least one candidate subnetwork based on the trained accuracy predictor.

[0160] It should be noted that, in the embodiments of this application, the accuracy of at least one candidate sub-network can be calculated based on the accuracy predictor after training, and the search can be performed based on the accuracy of the candidate sub-networks.

[0161] Step 905: Search to obtain the target subnetwork

[0162] It should be noted that the embodiments of this application can search for the target subnetwork based on satisfying the maximum accuracy constraint. That is, the candidate subnetwork corresponding to the maximum accuracy among at least one candidate subnetwork is taken as the target subnetwork. Ultimately, the target subnetwork satisfies the performance constraints of computing power and accuracy, realizing the possibility of searching for a subnetwork with both excellent computing power and accuracy.

[0163] VI. An exemplary description of a neural network search method

[0164] Based on the above description, an exemplary description of a neural network search method according to an embodiment of this application will be provided below.

[0165] like Figure 4 As shown, Figure 4 This is a flowchart illustrating a neural network search method according to an embodiment of this application. The method may specifically include the following steps:

[0166] S410. Obtain the first subnetwork in the supernet of the first neural network architecture search.

[0167] S420. Construct a supernet for searching the second neural network architecture based on the first subnetwork.

[0168] S430. Use the supernet of the first neural network architecture search as the teacher network in knowledge distillation to train the supernet of the second neural network architecture search.

[0169] S440. The target subnetwork is obtained by searching the supernetwork of the trained second neural network architecture with performance constraints of joint computing power and accuracy.

[0170] The performance constraints for combined computing power and accuracy can include satisfying preset computing power constraints and satisfying maximum accuracy constraints.

[0171] It should be noted that "first neural network architecture search", "first sub-network", "second neural network architecture search", "local knowledge distillation", "target sub-network", etc. can be found in the relevant content and description above, and will not be repeated here.

[0172] As can be seen, this application constructs a second neural network architecture search supernet by using a first sub-network in a supernet searched by a first neural network architecture, enabling the second neural network architecture search supernet to have the same performance as the first sub-network. This suggests that if the first sub-network has the best network performance, the second neural network architecture search supernet may also have the best network performance. During the training of the second neural network architecture search supernet, knowledge distillation is used to treat the first neural network architecture search supernet as a teacher network to train the second neural network architecture search supernet, resulting in better network performance after training. From the trained second neural network architecture search supernet, a target subnet is searched using joint performance constraints of computational power and accuracy, ensuring that the target subnet meets the computational power and accuracy requirements, thus realizing the possibility of finding a subnet with both excellent computational power and accuracy.

[0173] In some possible implementations, the supernet for the search of the second neural network architecture based on the first subnetwork in S420 may include the following steps:

[0174] Dynamically adjust at least one of the following in the first sub-network: network width, network depth, convolutional kernel size, number of convolutional kernels, and feature map resolution to obtain a baseline model;

[0175] The baseline model is used as a supernet for the second neural network architecture search.

[0176] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, since the first sub-network has a fixed network width, network depth, convolution kernel size, number of convolution kernels and feature map resolution, the baseline model is obtained by dynamically adjusting at least one of the network width, network depth, convolution kernel size, number of convolution kernels and feature map resolution in the first sub-network, and the baseline model is used as the supernet for the second neural network architecture search.

[0177] As can be seen, because the embodiments of this application can dynamically adjust the first sub-network according to different needs and actual situations, the method of generating the supernet for the second neural network architecture search is diverse and flexible, and the supernet for the second neural network architecture search can have the same performance as the first sub-network. Thus, if the first sub-network has optimal network performance, the supernet for the second neural network architecture search may also have optimal network performance.

[0178] In some possible implementations, the process of using the supernet of the first neural network architecture search as the teacher network in knowledge distillation to train the supernet of the second neural network architecture search in S430 may include the following steps:

[0179] The supernet searched by the first neural network architecture is used as the teacher network in knowledge distillation.

[0180] At least one second subnetwork in the supernet searched by the second neural network architecture is used as the student network in knowledge distillation.

[0181] With the objective of minimizing the loss value of the loss function between the student network and the teacher network for the first input data, at least one second sub-network is trained to train the supernet for searching the second neural network architecture.

[0182] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, in the training process of the supernet for the second neural network architecture search in this embodiment, it is necessary to sequentially collect a subnetwork (i.e., a second subnetwork) from the network of the second neural network architecture search and use this subnetwork as a student network for training, until a certain number of subnetworks (i.e., at least one second subnetwork) in the supernet for the second neural network architecture search are trained to complete the training of the supernet for the second neural network architecture search. Since this embodiment does not need to train all subnetworks in the supernet for the second neural network architecture search, but only a certain number of subnetworks, it saves training time and improves training efficiency.

[0183] Furthermore, by combining knowledge distillation to use the supernet of the first neural network architecture search as the teacher network to train the supernet of the second neural network architecture search, and by minimizing the loss value of the loss function between the output results of the student network and the teacher network for the first input data, the trained supernet of the second neural network architecture search can achieve better network performance.

[0184] In some possible implementations, aiming to minimize the loss function between the student network and the teacher network's outputs on the first input data can include the following steps:

[0185] Obtain the first loss value of the loss function between the output of the j-th layer in the student network and the j-th layer in the teacher network for the first input data, where j represents the layer number;

[0186] Obtain the second loss value of the loss function between the output of the j-th layer in the student network and the j-th layer in the ground truth for the first input data;

[0187] The product between the first loss value and the regularization term coefficient is calculated to obtain the third loss value, which is determined by the first loss value, the second loss value, and the weight parameters of the j-th layer in the student network.

[0188] The objective is to minimize the sum of the second and third loss values.

[0189] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, the embodiments of this application consider the value of the loss function between the output results of the j-th layer in the student network and the j-th layer in the teacher network for the first input data (i.e., the first loss value), and the value of the loss function between the output results of the j-th layer in the student network and the j-th layer in the ground truth for the first input data (i.e., the second loss value). The third loss value is calculated by using the first loss value and the regularization term coefficient. This ensures that when training the supernet for the second neural network architecture search with the goal of minimizing the sum of the second and third loss values, the network performance of the supernet trained for the second neural network architecture search is better, such as higher precision and higher accuracy.

[0190] In some possible implementations, the regularization term coefficients are determined by the first loss value, the second loss value, and the weight parameters of the j-th layer in the student network, and may include the following steps:

[0191] The regularization coefficient a(i,j) is:

[0192] a(i,j)=cos(dR(i,j) / dw(i,j),dL(i,j) / dw(i,j));

[0193] Where i represents the subnetwork number of the student network in the supernet of the second neural network architecture search; R(i,j) represents the first loss value; dR(i,j) represents the gradient of R(i,j); w(i,j) represents the weight parameter of the j-th layer in the student network; dw(i,j) represents the gradient of w(i,j); L(i,j) represents the second loss value; and dL(i,j) represents the gradient of L(i,j).

[0194] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, the embodiments of this application can measure the vector difference between two gradient optimization vectors (i.e., dR(i,j) / dw(i,j) and dL(i,j) / dw(i,j)) through cosine calculation, thereby improving the accuracy and precision of network optimization.

[0195] In some possible implementations, the target subnetwork is searched from the supernetwork of the trained second neural network architecture under performance constraints of joint computational power and accuracy, including:

[0196] Obtain the accuracy predictor and train the accuracy predictor;

[0197] From the supernet searched by the trained second neural network architecture, subnetworks are searched to meet the preset computing power constraints, and at least one candidate subnetwork is obtained.

[0198] Calculate the accuracy of at least one candidate subnetwork based on the trained accuracy predictor;

[0199] The candidate subnetwork corresponding to the maximum accuracy of at least one candidate subnetwork is taken as the target subnetwork.

[0200] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, since the performance constraints of combined computing power and accuracy include satisfying the preset computing power constraint and satisfying the maximum accuracy constraint, we can first satisfy the preset computing power constraint to search for at least one candidate subnetwork, and then satisfy the maximum accuracy constraint to search for the target subnetwork from at least one candidate subnetwork. This allows the target subnetwork to be a subnetwork that satisfies both the preset requirements for computing power and accuracy, thus achieving the possibility of having both excellent computing power and accuracy in the target subnetwork.

[0201] In some possible implementations, the first neural network architecture search includes a differentiable architecture search;

[0202] The second neural network architecture search involves training a network specifically for efficient deployment (Once-For-All).

[0203] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, since the search for differentiable architectures aims at accuracy, and the computational power of the resulting subnetworks is also relatively large, it ensures that the accuracy of the first subnetwork is very high when the first neural network architecture search includes the search for differentiable architectures. Thus, if the first subnetwork has optimal accuracy, the supernetwork from the second neural network architecture search also has the potential to have optimal accuracy.

[0204] Furthermore, since Once-For-All decouples the training and search processes, when the search of the first neural network architecture includes Once-For-All, the target subnetwork searched from the trained Once-For-All supernet does not need to be retrained.

[0205] VII. Exemplary Description of a Neural Network Search Device

[0206] The foregoing mainly describes the solutions of the embodiments of this application from the perspective of the method execution process. It is understood that, in order to achieve the above functions, the device includes hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should recognize that the methods, functions, modules, units, or steps described in conjunction with the embodiments provided herein can be implemented in hardware or a combination of hardware and computer software. Whether a method, function, module, unit, or step is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described methods, functions, modules, units, or steps for each specific application, but such implementation should not be considered beyond the scope of this application.

[0207] This application embodiment can divide an electronic device into functional units / modules based on the above method examples. For example, each function can be divided into its own functional unit / module, or two or more functions can be integrated into one functional unit / module. The integrated functional unit / module can be implemented in hardware or software. It should be noted that the division of functional units / modules in this application embodiment is illustrative and only represents a logical functional division; in actual implementation, there may be other division methods.

[0208] When using integrated units, Figure 5 This is a functional unit block diagram of a neural network search device according to an embodiment of this application. The neural network search device 500 specifically includes: an acquisition unit 510, a construction unit 520, a training unit 530, and a search unit 540.

[0209] It should be noted that the acquisition unit 510 can be a module unit for processing signals, data, or information; the construction unit 520 can be a module unit for processing signals, data, or information; the training unit 530 can be a module unit for processing signals, data, or information; and the search unit 540 can be a module unit for processing signals, data, or information.

[0210] In some possible implementations, the acquisition unit 510, the construction unit 520, the training unit 530, and the search unit 540 can be integrated into one unit or into different units.

[0211] For example, the acquisition unit 510, construction unit 520, training unit 530, and search unit 540 can be integrated into the processing unit. The processing unit can be a processor or controller, such as a central processing unit (CPU), a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processing unit can also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0212] In some possible implementations, the neural network search device 500 may also include a storage unit for storing computer programs or instructions executed by the neural network search device 500.

[0213] For example, the storage unit could be a memory.

[0214] In some possible implementations, the neural network search device 500 can be a chip / chip module / processor, etc.

[0215] In specific implementation, the neural network search device 500 is used to perform the steps described in the above method embodiments. A detailed explanation follows.

[0216] The acquisition unit 510 is used to acquire the first sub-network in the supernet searched by the first neural network architecture.

[0217] Construction unit 520 is used to construct a supernet for the second neural network architecture search based on the first subnetwork;

[0218] Training unit 530 is used to train the supernet of the second neural network architecture search by using the supernet of the first neural network architecture search as the teacher network in knowledge distillation.

[0219] Search unit 540 is used to search for the target subnetwork from the supernet of the trained second neural network architecture with performance constraints of joint computing power and accuracy.

[0220] It should be noted that the specific implementation of each operation performed by the neural network search device 500 can be found in the above description. Figure 4 The corresponding descriptions of the method embodiments shown will not be repeated here.

[0221] As can be seen, this application constructs a second neural network architecture search supernet by using a first sub-network in a supernet searched by a first neural network architecture, enabling the second neural network architecture search supernet to have the same performance as the first sub-network. This suggests that if the first sub-network has the best network performance, the second neural network architecture search supernet may also have the best network performance. During the training of the second neural network architecture search supernet, knowledge distillation is used to treat the first neural network architecture search supernet as a teacher network to train the second neural network architecture search supernet, resulting in better network performance after training. From the trained second neural network architecture search supernet, a target subnet is searched using joint performance constraints of computational power and accuracy, ensuring that the target subnet meets the computational power and accuracy requirements, thus realizing the possibility of finding a subnet with both excellent computational power and accuracy.

[0222] In some possible implementations, in terms of constructing the supernet for the search of the second neural network architecture based on the first subnetwork, the building unit 520 is used for:

[0223] Dynamically adjust at least one of the following in the first sub-network: network width, network depth, convolutional kernel size, number of convolutional kernels, and feature map resolution to obtain a baseline model;

[0224] The baseline model is used as a supernet for the second neural network architecture search.

[0225] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, since the first sub-network has a fixed network width, network depth, convolution kernel size, number of convolution kernels and feature map resolution, the baseline model is obtained by dynamically adjusting at least one of the network width, network depth, convolution kernel size, number of convolution kernels and feature map resolution in the first sub-network, and the baseline model is used as the supernet for the second neural network architecture search.

[0226] As can be seen, because the embodiments of this application can dynamically adjust the first sub-network according to different needs and actual situations, the method of generating the supernet for the second neural network architecture search is diverse and flexible, and the supernet for the second neural network architecture search can have the same performance as the first sub-network. Thus, if the first sub-network has optimal network performance, the supernet for the second neural network architecture search may also have optimal network performance.

[0227] In some possible implementations, in training the supernet of the first neural network architecture search as a teacher network in knowledge distillation to train the supernet of the second neural network architecture search, the training unit 530 is used for:

[0228] The supernet searched by the first neural network architecture is used as the teacher network in knowledge distillation.

[0229] At least one second subnetwork in the supernet searched by the second neural network architecture is used as the student network in knowledge distillation.

[0230] With the objective of minimizing the loss value of the loss function between the student network and the teacher network for the first input data, at least one second sub-network is trained to train the supernet for searching the second neural network architecture.

[0231] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, in the training process of the supernet for the second neural network architecture search in this embodiment, it is necessary to sequentially collect a subnetwork (i.e., a second subnetwork) from the network of the second neural network architecture search and use this subnetwork as a student network for training, until a certain number of subnetworks (i.e., at least one second subnetwork) in the supernet for the second neural network architecture search are trained to complete the training of the supernet for the second neural network architecture search. Since this embodiment does not need to train all subnetworks in the supernet for the second neural network architecture search, but only a certain number of subnetworks, it saves training time and improves training efficiency.

[0232] Furthermore, by combining knowledge distillation to use the supernet of the first neural network architecture search as the teacher network to train the supernet of the second neural network architecture search, and by minimizing the loss value of the loss function between the output results of the student network and the teacher network for the first input data, the trained supernet of the second neural network architecture search can achieve better network performance.

[0233] In some possible implementations, the training unit 530 is used to: minimize the value of the loss function between the outputs of the student network and the teacher network on the first input data.

[0234] Obtain the first loss value of the loss function between the output of the j-th layer in the student network and the j-th layer in the teacher network for the first input data, where j represents the layer number;

[0235] Obtain the second loss value of the loss function between the output of the j-th layer in the student network and the j-th layer in the ground truth for the first input data;

[0236] The product between the first loss value and the regularization term coefficient is calculated to obtain the third loss value, which is determined by the first loss value, the second loss value, and the weight parameters of the j-th layer in the student network.

[0237] The objective is to minimize the sum of the second and third loss values.

[0238] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, the embodiments of this application consider the value of the loss function between the output results of the j-th layer in the student network and the j-th layer in the teacher network for the first input data (i.e., the first loss value), and the value of the loss function between the output results of the j-th layer in the student network and the j-th layer in the ground truth for the first input data (i.e., the second loss value). The third loss value is calculated by using the first loss value and the regularization term coefficient. This ensures that when training the supernet for the second neural network architecture search with the goal of minimizing the sum of the second and third loss values, the network performance of the supernet trained for the second neural network architecture search is better, such as higher precision and higher accuracy.

[0239] In some possible implementations, the regularization term coefficients are determined by the first loss value, the second loss value, and the weight parameters of the j-th layer in the student network, including:

[0240] The regularization coefficient a(i,j) is:

[0241] a(i,j)=cos(dR(i,j) / dw(i,j),dL(i,j) / dw(i,j));

[0242] Where i represents the subnetwork number of the student network in the supernet of the second neural network architecture search; R(i,j) represents the first loss value; dR(i,j) represents the gradient of R(i,j); w(i,j) represents the weight parameter of the j-th layer in the student network; dw(i,j) represents the gradient of w(i,j); L(i,j) represents the second loss value; and dL(i,j) represents the gradient of L(i,j).

[0243] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, the embodiments of this application can measure the vector difference between two gradient optimization vectors (i.e., dR(i,j) / dw(i,j) and dL(i,j) / dw(i,j)) through cosine calculation, thereby improving the accuracy and precision of network optimization.

[0244] In some possible implementations, the search unit 540 is used to: search for the target subnetwork from the supernet searched from the trained second neural network architecture with performance constraints of joint computing power and accuracy.

[0245] Obtain the accuracy predictor and train the accuracy predictor;

[0246] From the supernet searched by the trained second neural network architecture, subnetworks are searched to meet the preset computing power constraints, and at least one candidate subnetwork is obtained.

[0247] Calculate the accuracy of at least one candidate subnetwork based on the trained accuracy predictor;

[0248] The candidate subnetwork corresponding to the maximum accuracy of at least one candidate subnetwork is taken as the target subnetwork.

[0249] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, since the performance constraints of combined computing power and accuracy include satisfying the preset computing power constraint and satisfying the maximum accuracy constraint, we can first satisfy the preset computing power constraint to search for at least one candidate subnetwork, and then satisfy the maximum accuracy constraint to search for the target subnetwork from at least one candidate subnetwork. This allows the target subnetwork to be a subnetwork that satisfies both the preset requirements for computing power and accuracy, thus achieving the possibility of having both excellent computing power and accuracy in the target subnetwork.

[0250] In some possible implementations, the first neural network architecture search includes a differentiable architecture search;

[0251] The second neural network architecture search involves training a network specifically for efficient deployment (Once-For-All).

[0252] It should be noted that, in conjunction with the content in "2. Implementation Principle" above, since the search for differentiable architectures aims at accuracy, and the computational power of the resulting subnetworks is also relatively large, it ensures that the accuracy of the first subnetwork is very high when the first neural network architecture search includes the search for differentiable architectures. Thus, if the first subnetwork has optimal accuracy, the supernetwork from the second neural network architecture search also has the potential to have optimal accuracy.

[0253] Furthermore, since Once-For-All decouples the training and search processes, when the search of the first neural network architecture includes Once-For-All, the target subnetwork searched from the trained Once-For-All supernet does not need to be retrained.

[0254] 8. Exemplary Description of an Electronic Device

[0255] The following is a schematic diagram of the structure of an electronic device according to an embodiment of this application, such as... Figure 6 As shown. The electronic device 600 includes a processor 610, a memory 620, and at least one communication bus for connecting the processor 610 and the memory 620.

[0256] In some possible implementations, processor 610 may be one or more central processing units (CPUs). If processor 610 is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Memory 620 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and memory 620 is used to store computer programs or instructions.

[0257] In some possible implementations, the electronic device 600 also includes a communication interface for receiving and sending data.

[0258] In some possible implementations, the processor 610 in the electronic device 600 is used to execute a computer program or instruction 621 stored in the memory 620 to perform the following steps:

[0259] Obtain the first subnetwork in the supernet searched by the first neural network architecture;

[0260] A supernet is searched by constructing a second neural network architecture based on the first subnetwork;

[0261] The supernet searched by the first neural network architecture is used as the teacher network in knowledge distillation to train the supernet searched by the second neural network architecture architecture.

[0262] The target subnetwork is obtained by searching the supernetwork of the trained second neural network architecture with performance constraints of joint computing power and accuracy.

[0263] It should be noted that the specific implementation of each operation performed by the electronic device 600 can be found in the above description. Figure 4 The corresponding descriptions of the method embodiments shown will not be repeated here.

[0264] As can be seen, this application constructs a second neural network architecture search supernet by using a first sub-network in a supernet searched by a first neural network architecture, enabling the second neural network architecture search supernet to have the same performance as the first sub-network. This suggests that if the first sub-network has the best network performance, the second neural network architecture search supernet may also have the best network performance. During the training of the second neural network architecture search supernet, knowledge distillation is used to treat the first neural network architecture search supernet as a teacher network to train the second neural network architecture search supernet, resulting in better network performance after training. From the trained second neural network architecture search supernet, a target subnet is searched using joint performance constraints of computational power and accuracy, ensuring that the target subnet meets the computational power and accuracy requirements, thus realizing the possibility of finding a subnet with both excellent computational power and accuracy.

[0265] IX. Other Exemplary Descriptions

[0266] This application also provides a computer-readable storage medium storing a computer program or instructions that, when executed by a processor, implement the steps described in the above embodiments.

[0267] This application also provides a computer program product, including a computer program or instructions, wherein the computer program or instructions, when executed by a processor, implement the steps described in the above embodiments. For example, the computer program product may be a software installation package.

[0268] In addition, computer program products should be understood as software products that primarily implement the technical solutions of this application through computer programs or instructions.

[0269] It should be noted that, for the sake of simplicity, the above embodiments are all described as a series of actions. Those skilled in the art should understand that this application is not limited to the described order of actions, as some steps in the embodiments of this application can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions, steps, modules, or units involved are not necessarily essential to the embodiments of this application.

[0270] In the above embodiments, the descriptions of each embodiment in this application have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0271] Those skilled in the art should understand that the functions of the methods, steps, or related modules / units described in the embodiments of this application can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product, or by a processor executing computer program instructions. The computer program product includes at least one computer program instruction, which can be composed of corresponding software modules. These software modules can be stored in RAM, flash memory, ROM, EPROM, EEPROM, registers, hard disk, portable hard disk, read-only optical disc (CD-ROM), or any other form of storage medium well known in the art. The computer program instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer program instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media, or semiconductor media (e.g., SSDs).

[0272] The modules / units included in the various devices or products described in the above embodiments can be software modules / units, hardware modules / units, or a combination of software and hardware modules / units. For example, for devices or products applied to or integrated into a chip, all of their modules / units can be implemented using hardware methods such as circuits; or, some of their modules / units can be implemented using software programs that run on a processor integrated within the chip, while other (if any) modules / units can be implemented using hardware methods such as circuits. The same principle applies to devices or products applied to or integrated into chip modules, or devices or products applied to or integrated into terminals.

[0273] The above detailed embodiments further illustrate the purpose, technical solution, and beneficial effects of the embodiments of this application. It should be understood that the above are merely specific implementations of the embodiments of this application and are not intended to limit the protection scope of the embodiments of this application. Any modifications, equivalent substitutions, improvements, etc., made based on the technical solutions of the embodiments of this application should be included within the protection scope of the embodiments of this application.

Claims

1. A neural network search method, characterized by, The method comprises the following steps: obtaining a first subnetwork in a super network of a first neural network architecture search; constructing a super network of a second neural network architecture search according to the first subnetwork; taking the super network of the first neural network architecture search as a teacher network in knowledge distillation; sequentially taking at least one second subnetwork in the super network of the second neural network architecture search as a student network in knowledge distillation; obtaining a first loss value of a loss function between an output result of a jth layer in the student network and an output result of the jth layer in the teacher network for first input data, j representing a layer number; obtaining a second loss value of a loss function between an output result of the jth layer in the student network and an output result of the jth layer in ground truth for the first input data; calculating a product between the first loss value and a regularization term coefficient to obtain a third loss value, the regularization term coefficient being determined by the first loss value, the second loss value and a weight parameter of the jth layer in the student network; training at least one second subnetwork to train the super network of the second neural network architecture search, aiming at minimizing a sum of the second loss value and the third loss value, the first input data comprising images; and searching a target subnetwork from the trained super network of the second neural network architecture search under a performance constraint of joint computing power and accuracy.

2. The method of claim 1, wherein, The step of constructing a super network of a second neural network architecture search according to the first subnetwork comprises the following steps: dynamically adjusting at least one of a network width, a network depth, a convolution kernel size, a convolution kernel number and a feature map resolution in the first subnetwork to obtain a baseline model; taking the baseline model as a super network of a second neural network architecture search.

3. The method of claim 1, wherein, The step of determining the regularization term coefficient by the first loss value, the second loss value and the weight parameter of the jth layer in the student network comprises the following steps: The regularization term coefficient a(i,j) is: a(i,j)=cos(dR(i,j) / dw(i,j), dL(i,j) / dw(i,j)); wherein i represents a subnetwork number of the student network in the super network of the second neural network architecture search; R(i,j) represents the first loss value; dR(i,j) represents a gradient of R(i,j); w(i,j) represents the weight parameter of the jth layer in the student network; dw(i,j) represents a gradient of w(i,j); L(i,j) represents the second loss value; and dL(i,j) represents a gradient of L(i,j).

4. The method of claim 1, wherein, The step of searching a target subnetwork from the trained super network of the second neural network architecture search under a performance constraint of joint computing power and accuracy comprises the following steps: obtaining an accuracy predictor and training the accuracy predictor; searching a subnetwork from the trained super network of the second neural network architecture search to meet a preset computing power constraint condition to obtain at least one candidate subnetwork; calculating respective accuracies of the at least one candidate subnetwork according to the trained accuracy predictor. The candidate sub-network corresponding to the maximum value in the precision of each of the at least one candidate sub-network is taken as a target sub-network.

5. The method according to any one of claims 1 to 4, characterized in that, The first neural network architecture search comprises a differentiable architecture search; The second neural network architecture search comprises training a network and is specially used for efficient deployment.

6. A neural network searching apparatus, comprising: Comprise: An acquisition unit is configured to acquire a first sub-network in a super-network of a first neural network architecture search; A construction unit is configured to construct a super-network of a second neural network architecture search according to the first sub-network; A training unit is configured to take the super-network of the first neural network architecture search as a teacher network in knowledge distillation; take at least one second sub-network in the super-network of the second neural network architecture search as a student network in knowledge distillation in turn; acquire a first loss value of a loss function between an output result of a jth layer in the student network and an output result of the jth layer in the teacher network for first input data, j representing a layer number; acquire a second loss value of the loss function between the output result of the jth layer in the student network and an output result of a jth layer in ground truth for the first input data; calculate a product between the first loss value and a regularization term coefficient to obtain a third loss value, the regularization term coefficient being determined by the first loss value, the second loss value and a weight parameter of the jth layer in the student network; train at least one second sub-network to realize training of the super-network of the second neural network architecture search, with a target of minimizing a sum of the second loss value and the third loss value, the first input data comprising an image; A searching unit is configured to search a target sub-network from the trained super-network of the second neural network architecture search with a performance constraint of joint computing power and precision.

7. An electronic device comprising a processor, a memory, and a computer program or instructions stored on the memory, wherein, The processor executes the computer program or instructions to implement the steps of the method of any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The computer program or instructions stored in the computer readable storage medium are executed to implement the steps of the method of any one of claims 1-5.

Images