Neural network structure search method, device, electronic device and storage medium

Abstract

The embodiments of the present application provide a neural network structure search method, device, electronic device and storage medium. The neural network structure search method includes: obtaining a twin super network constructed based on a preset search space, the twin super network including a first super network and a second super network; obtaining a loss value based on the output vector of the first super network, the output vector of the second super network and a preset loss function, optimizing the weight parameters of the twin super network based on the loss value to obtain a candidate network structure; performing a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure; adjusting the weight parameters of the twin super network based on the performance score, and returning to the step of obtaining the candidate network structure until the preset optimization termination condition is met; determining the target network structure from multiple candidate network structures based on the performance score of each candidate network structure. The embodiments of the present application improve the accuracy of structure search.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a neural network structure search method, device, electronic device, and storage medium. Background Art

[0002] With the continuous development of deep learning technology, NAS (Neural Architecture Search) technology has become a research hotspot. NAS technology automatically searches for network structures in a predefined search space to obtain the optimal neural network model structure.

[0003] Currently, neural network architecture search is typically a supervised search process, meaning it relies on an external network model trained with labels to provide supervisory information. Because the search is based on the supervised information provided by the network model, the target network found is more likely to have the same network structure as the aforementioned network model. This can lead to structural bias in the search results, which in turn affects search accuracy. Summary of the Invention

[0004] In view of this, embodiments of the present application provide a neural network structure search method, device, electronic device, and computer storage medium to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of an embodiment of the present application, a neural network structure search method is provided, comprising:

[0006] Obtain a twin supernetwork constructed based on a preset search space, wherein the twin supernetwork includes a first supernetwork and a second supernetwork; the second supernetwork has the same network structure as the first supernetwork, and the network weight of the second supernetwork is obtained based on the network weight of the first supernetwork;

[0007] Obtaining a loss value according to the output vector of the first supernetwork, the output vector of the second supernetwork, and a preset loss function, and optimizing the weight parameters of the twin supernetwork based on the loss value to obtain a candidate network structure;

[0008] Performing a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure;

[0009] Based on the performance score, the weight parameters of the twin super network are adjusted, and the loss value obtained according to the output vector of the first super network, the output vector of the second super network and the preset loss function is returned, and the weight parameters of the twin super network are optimized based on the loss value to obtain a candidate network structure until a preset optimization termination condition is met;

[0010] According to the performance scores of the candidate network structures, the target network structure is determined from multiple candidate network structures.

[0011] According to a second aspect of an embodiment of the present application, a neural network structure search device is provided, comprising:

[0012] A twin supernetwork acquisition module, configured to acquire a twin supernetwork constructed based on a preset search space, wherein the twin supernetwork includes a first supernetwork and a second supernetwork; the second supernetwork has the same network structure as the first supernetwork, and the network weight of the second supernetwork is obtained based on the network weight of the first supernetwork;

[0013] A candidate network structure obtaining module is used to obtain a loss value according to the output vector of the first super network, the output vector of the second super network and a preset loss function, and optimize the weight parameters of the twin super network based on the loss value to obtain a candidate network structure;

[0014] A performance evaluation module, configured to perform a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure;

[0015] A weight parameter adjustment module is used to adjust the weight parameters of the twin super network based on the performance score, and return the loss value obtained according to the output vector of the first super network, the output vector of the second super network and the preset loss function, and optimize the weight parameters of the twin super network based on the loss value to obtain a candidate network structure until a preset optimization termination condition is met;

[0016] The target network structure determination module is used to determine the target network structure from multiple candidate network structures based on the performance score of each candidate network structure.

[0017] According to a third aspect of an embodiment of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface and a communication bus, wherein the processor, the memory and the communication interface communicate with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the neural network structure search method described in the first aspect.

[0018] According to a fourth aspect of an embodiment of the present application, a computer storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the neural network structure search method as described in the first aspect is implemented.

[0019] According to the neural network structure search method provided in the embodiment of the present application, a twin supernetwork consisting of two supernetworks with the same structure is obtained, the loss value is calculated based on the output vector of the first supernetwork and the output vector of the second supernetwork, and the weight parameters of the twin supernetwork are optimized to obtain a candidate network structure; after obtaining multiple candidate network structures, the target network structure is determined based on the performance scores of each candidate network structure. In the embodiment of the present application, after obtaining two supernetworks, the supernetwork is optimized based on the output vectors of the two supernetworks to obtain the candidate network structure. That is to say, the above optimization process is carried out by self-supervised comparison between two supernetworks with the same structure, without relying on an external network model obtained based on label training to provide supervision information. Therefore, the problem of structural deviation in search results caused by relying on supervision information provided by an external network model is avoided, and the accuracy of structure search is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments recorded in the embodiments of the present application. For ordinary technicians in this field, other drawings can also be obtained based on these drawings.

[0021] Figure 1 This is a flowchart of a neural network structure search method according to the first embodiment of the present application;

[0022] Figure 2 A schematic diagram of the structure of a super network provided in an embodiment of the present application;

[0023] Figure 3 for Figure 1 A schematic diagram of an example scenario in the illustrated embodiment;

[0024] Figure 4 This is a flowchart of a neural network structure search method according to the second embodiment of the present application;

[0025] Figure 5 A schematic diagram of the connection relationship between candidate operators in the search space provided in an embodiment of the present application;

[0026] Figure 6 A schematic diagram of the super network blocks provided in an embodiment of the present application;

[0027] Figure 7 A schematic diagram of the calculation process of the loss value during the weight parameter optimization process of the twin super network module provided in an embodiment of the present application;

[0028] Figure 81 is a structural block diagram of a neural network structure search device according to the fourth embodiment of the present application;

[0029] Figure 9 This is a structural diagram of an electronic device according to the fifth embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the embodiments of the present application, all other embodiments obtained by ordinary technicians in this field should fall within the scope of protection of the embodiments of the present application.

[0031] The specific implementation of the embodiment of the present application is further explained below in conjunction with the accompanying drawings of the embodiment of the present application.

[0032] Example 1

[0033] Reference Figure 1 , Figure 1 This is a flowchart of the steps of a neural network structure search method according to Example 1 of the present application.

[0034] Specifically, the neural network structure search method provided in this embodiment includes the following steps:

[0035] Step 102: Acquire a twin supernetwork constructed based on a preset search space, where the twin supernetwork includes a first supernetwork and a second supernetwork.

[0036] The second super network has the same network structure as the first super network, and the network weight of the second super network is obtained based on the network weight of the first super network.

[0037] Before conducting a network structure search, a variety of basic units that can be used to construct different network structures are usually pre-set. These basic units are also called candidate operators, such as convolution candidate operators for convolution operations, self-attention candidate operators, etc. The connection relationships between the above candidate operators are combined to obtain the corresponding search space.

[0038] The super network in the embodiment of the present application includes all optional neural network structures in the search space. In other words, the super network is multi-path, where each path corresponds to an optional neural network structure in the search space. Figure 2 , Figure 2This is a structural diagram of a hypernetwork in an embodiment of the present application, wherein the hypernetwork includes 6 layers, each layer includes 3 branches (a circle in the figure is a branch), and a branch represents a candidate operator. By selecting a branch in each layer and connecting and stacking it with the branches selected from other layers, a path can be formed. This path corresponds to a neural network structure in the search space.

[0039] In the embodiments of the present application, two hypernetworks with identical structures are constructed, and the network weights of the two hypernetworks are correlated. Specifically, the network weights of the second hypernetwork are derived based on the network weights of the first hypernetwork. For example, the network weights of the second hypernetwork can be the same as those of the first hypernetwork, or the network weights of the second hypernetwork can be derived by performing a preset transformation on the network weights of the first hypernetwork. The preset transformation can be set according to actual needs, such as a sliding average transformation, scaling, etc.

[0040] Step 104: Obtain a loss value based on the output vector of the first supernetwork, the output vector of the second supernetwork, and a preset loss function, and optimize the weight parameters of the twin supernetwork based on the loss value to obtain a candidate network structure.

[0041] The preset loss function can be a function with the output vector of the first supernetwork and the output vector of the second supernetwork as independent variables. For example, the similarity between the output vector of the first supernetwork and the output vector of the second supernetwork can be directly used as the loss function, or a certain transformation operation can be performed on the above similarity, and the transformation result can be used as the loss function. The higher the above similarity, the smaller the loss value corresponding to the loss function; the output vector of the first supernetwork or the output vector of the second supernetwork can also be transformed, and the similarity between the transformed output vectors can be used as the loss function, and so on. In the embodiments of the present application, the specific form of the loss function is not limited.

[0042] In this step, a loss value is obtained based on the output vector of the first supernetwork, the output vector of the second supernetwork and the above-mentioned loss function, and the weight parameters of the first supernetwork and the second supernetwork are optimized based on the loss value, so as to determine a path among the multiple paths of the first supernetwork, and the neural network structure corresponding to the path is used as a candidate network structure.

[0043] Step 106: Perform a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure.

[0044] Since the input samples and network structures (paths in the hypernetwork) used for optimization are sampled, the loss obtained in the self-supervised comparative optimization phase does not necessarily represent the performance and convergence of the network structure. Therefore, it is necessary to evaluate the performance of the candidate network structure.

[0045] The performance score is used to characterize the performance of the candidate network structure, such as accuracy, convergence, etc. The higher the performance score, the better the performance of the candidate network structure.

[0046] After obtaining the candidate network structures, it is necessary to perform performance evaluation on each candidate network structure separately, so as to subsequently select a certain number of candidate network structures based on the performance scores of each candidate network structure as the final target network structure.

[0047] When optimizing the twin supernetwork, the embodiment of the present application performs the optimization through self-supervised comparison. Correspondingly, in the performance evaluation process, an unsupervised performance evaluation method can also be used. Commonly used unsupervised performance evaluation methods include using the accuracy of the preceding task to perform performance evaluation on the candidate network structure, or performing performance evaluation on the candidate network structure based on the convergence metric of the angle metric, and so on. In addition, it is also possible to: for each candidate network structure, calculate the output vector of the candidate network structure and the output vectors of other network structures in the first supernetwork, and then obtain the performance score of the candidate network structure based on the similarity between the output vector of the candidate network structure and the output vectors of the other network structures. The higher the similarity, the higher the performance score of the candidate network structure. Specifically, the performance evaluation method can be set according to actual needs and is not limited here.

[0048] Step 108: Based on the performance score, adjust the weight parameters of the twin super network, return the loss value obtained based on the output vector of the first super network, the output vector of the second super network and the preset loss function, optimize the weight parameters of the twin super network based on the loss value, and obtain the candidate network structure until the preset optimization termination condition is met.

[0049] Based on the performance scores of the candidate network structures, the adjustment direction of the weight parameters of the twin super network can be determined. Then, based on the twin super network after the weight parameters are adjusted, the optimization and performance evaluation process of steps 104 to 106 is performed again until the preset optimization termination conditions are met. At this point, multiple candidate network structures and the performance scores of each candidate network structure can be obtained.

[0050] The optimization termination condition in this step can be set according to actual needs. For example, the optimization termination condition can be determined to be satisfied when the number of optimizations reaches a preset number threshold; or the optimization termination condition can be determined to be satisfied when the loss value obtained based on step 104 is less than a preset loss value threshold, etc. In the embodiments of the present application, the specific content of the optimization termination condition is not limited.

[0051] Step 110 : determining a target network structure from a plurality of candidate network structures according to the performance scores of the candidate network structures.

[0052] Specifically, for example, a preset number of candidate network structures may be selected as the target network structure in descending order of performance scores.

[0053] See also Figure 3 , Figure 3 This is a schematic diagram of the scene corresponding to the first embodiment of the present application. Figure 3 The schematic diagram shown in the figure illustrates an embodiment of the present application using a specific scenario as an example:

[0054] A twin supernetwork including a first supernetwork and a second supernetwork is obtained, wherein the structures of the first supernetwork and the second supernetwork are the same: both have 6 layers, each layer has 3 branches, and the weight parameters of the second supernetwork are obtained based on the weight parameters of the first supernetwork; the input X is input into the first supernetwork to obtain the output vector A of the first supernetwork, and the input X' is input into the second supernetwork to obtain the output vector B of the second supernetwork. After the loss value is obtained based on A, B and a preset loss function, the weight parameters of the twin supernetwork can be optimized based on the loss value to obtain a candidate network structure; a performance evaluation is performed on the obtained candidate network structure to obtain a performance score of the candidate network structure, and the weight parameters of the twin supernetwork are adjusted again based on the performance score, thereby executing the above-mentioned weight parameter optimization process again until the optimization termination condition is met. At this point, 4 candidate network structures are obtained: a, b, c and d, and the performance scores of each candidate network structure are: the performance score of a is 90 points, the performance score of b is 85, the performance score of c is 80, and the performance score of d is 75 points (in the embodiment of the present application, the specific representation of the performance score is not limited, Figure 3 Finally, according to the order of performance scores from high to low, a and b are determined as the searched target network structures.

[0055] In an embodiment of the present application, after obtaining two supernetworks, the supernetworks are optimized based on the output vectors of the two supernetworks to obtain a candidate network structure. That is to say, the above optimization process is carried out by self-supervised comparison between two supernetworks with the same structure, without relying on an external network model obtained based on label training to provide supervision information. Therefore, the problem of structural deviation in search results due to reliance on supervision information provided by an external network model is avoided, thereby improving the accuracy of structural search.

[0056] The neural network structure search method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, etc.

[0057] Example 2

[0058] Reference Figure 4, Figure 4 This is a flowchart of the steps of a neural network structure search method according to Example 2 of the present application.

[0059] Specifically, the neural network structure search method provided in this embodiment includes the following steps:

[0060] Step 402: Acquire a twin supernetwork constructed based on a preset search space, where the twin supernetwork includes a first supernetwork and a second supernetwork.

[0061] The second super network has the same network structure as the first super network, and the network weight of the second super network is obtained by performing exponential sliding average processing on the network weight of the first super network.

[0062] Furthermore, for the search space in the embodiment of the present application, its convolution candidate operators may include: the residual module in the ResNet network; the candidate operators of the preset search space may also include: a self-attention candidate operator composed of a depth-wise separable convolution module and an NLBlock module, or a self-attention candidate operator obtained by replacing the relative position encoding branch in the BoTBlock module with a depth-wise separable convolution module.

[0063] At present, in the conventional search space, BoTBlock module or NLBlock is usually used as the candidate self-attention operator. However, the computational complexity of the BoTBlock module is relatively high, mainly because of the relative position encoding branch, which is calculated separately by multiplying with the query; as for the NLBlock module, it lacks the relative position encoding branch, and position encoding is crucial to the performance of the self-attention module. Therefore, in the above conventional search space, when the BoTBlock module is used as the candidate self-attention operator, although the performance of the self-attention module can be guaranteed, the computational complexity is relatively high; when the NLBlock module is used as the candidate self-attention operator, although the computational complexity is low, due to the lack of the position encoding branch, the performance is poor.

[0064] In the search space in the embodiment of the present application, the self-attention candidate operator is composed of a depthwise separable convolution module and an NLBlock module, or is obtained by replacing the relative position encoding branch in the BoTBlock module with the depthwise separable convolution module. Therefore, the computational complexity can be reduced while ensuring the computational performance.

[0065] In addition, as far as the current CNN network and Transformer network are concerned, the two have great differences in macroscopic structure. Specifically, CNN network usually uses multiple stages of different scales to process image data, while Transformer network does not change the spatial dimension of the data. Therefore, in order to enable the search space to cover the above two network structures with large differences in macroscopic structure, the embodiment of the present application adopts the following method based on the above candidate operators: Figure 5 The textile-like connection relationship shown in the figure connects each candidate operator to form a search space, where Figure 5 The search space in contains L layers. The circles in each layer represent different candidate operators, and the lines with arrows represent the connections between candidate operators in different layers. This search space can cover a variety of different neural network structures, such as ResNet, BoTNet, ViT, DeiT, T2T-ViT, etc.

[0066] Step 404 : Divide the first supernetwork into blocks to obtain a plurality of first supernetwork modules; and divide the second supernetwork into blocks to obtain a plurality of second supernetwork modules.

[0067] The first super network module corresponds to the second super network module one by one.

[0068] Since the search space has many layers, the overall search requires a large amount of search. To improve search efficiency, the search space can be decomposed into blocks, that is, the super network is divided into blocks. The weight parameters of each module are then optimized separately, and the candidate sub-network structure in each module is determined. This results in a candidate network structure for each super network. This candidate network structure is composed of the candidate sub-network structures in each module.

[0069] by Figure 6 For example, the first hypernetwork is decomposed into two first hypernetwork modules, each containing three layers. Correspondingly, the second hypernetwork is decomposed into two second hypernetwork modules, each also containing three layers. Each first hypernetwork module and its corresponding second hypernetwork module form a twin hypernetwork module.

[0070] Step 406: Obtain optimized samples and perform data enhancement processing on the optimized samples to obtain enhanced samples.

[0071] In order to reduce the impact of the differences in the input samples themselves on the optimization results, in the embodiment of the present application, data enhancement processing is performed on the same optimized sample, and convolution processing is performed to obtain the input data that are subsequently input into the first super network module and the second super network module respectively.

[0072] In the embodiment of the present application, any existing data enhancement method can be used to perform data enhancement on the optimized samples, and the specific method of data enhancement is not limited here.

[0073] Step 408: Perform convolution processing on the enhanced sample to obtain a first enhanced optimized sample and a second enhanced optimized sample.

[0074] For each path (each neural sub-network structure) in each twin super-network module, its input is the output of the previous adjacent twin super-network module. In order to optimize the weight parameters in each twin super-network module in parallel to improve the optimization efficiency, in the embodiment of the present application, for each twin super-network module, after obtaining the enhanced sample, the enhanced sample is convolved so that the enhanced optimized sample can have the same size as the output of the adjacent twin super-network module of the previous level of the twin super-network module.

[0075] Step 410: For each first super network module, input the first enhanced optimized sample into the first super network module to obtain a first output vector; input the second enhanced optimized sample into the corresponding second super network module to obtain a second output vector.

[0076] Step 412: Based on the first output vector and the second output vector, a preset loss function is used to obtain a loss value.

[0077] The preset loss function can be a function with the first output vector and the second output vector as independent variables. For example, the similarity between the first output vector and the second output vector can be directly used as the loss function, or the similarity can be transformed and the transformation result used as the loss function, where the higher the similarity, the smaller the loss value corresponding to the loss function. Alternatively, the first output vector or the second output vector can be transformed, and the similarity between the transformed output vectors can be used as the loss function, etc. In the embodiments of the present application, the specific form of the loss function is not limited.

[0078] Furthermore, in some embodiments, step 410 of inputting the first enhanced optimized sample into each first super-network module to obtain a first output vector may include:

[0079] For each first super network module, path sampling is performed on the first super network module according to a preset sampling strategy to obtain multiple first sampling sub-networks; the first enhanced optimization sample is input into each first sampling sub-network respectively to obtain a first output vector of each first sampling sub-network.

[0080] Correspondingly, inputting the second enhanced optimized sample into the corresponding second super network module to obtain a second output vector may include:

[0081] According to a preset sampling strategy, path sampling is performed on the second super network module corresponding to the first super network module to obtain multiple second sampling sub-networks; the second enhanced optimization sample is input into each second sampling sub-network respectively to obtain a second output vector of each second sampling sub-network.

[0082] Correspondingly, step 412 may include:

[0083] Calculate the mean of all second output vectors as the full path mean;

[0084] For each first sampling subnetwork, subtract the first output vector of the first sampling subnetwork from the full path mean, as the difference value corresponding to the first sampling subnetwork;

[0085] A loss value is obtained based on the sum of the difference values ​​corresponding to each first sampling sub-network.

[0086] For ease of understanding, the following Figure 7 , explaining the above steps 406 to 412:

[0087] For a specific twin super network module, after obtaining the optimized sample image data, the image data is enhanced to obtain X, and then convolution processing is performed on X to obtain the first enhanced optimized sample X1 and the second enhanced optimized sample X2; X1 is input into the first super network module in the twin super network module, and the first output vector Y1 of a1 is output through the first sampling sub-network a1; X2 is input into the second sampling sub-networks b1, b2, b3 and b4 of the second super network module in the twin super network module, and the second output vectors y1, y2, y3 and y4 are output respectively. The mean of y1, y2, y3 and y4 is calculated to obtain the full path mean y0. For a1, the difference between Y1 and y0 is used as the difference La1 corresponding to a1; and so on, the other first sampling sub-networks a2, a3 and a4 are obtained ( Figure 7 The corresponding differences La2, La3 and La4 (not shown) are Figure 7 and then based on the sum of La1, La2, La3 and La4, obtain the final loss value, for example: directly taking the sum of La1, La2, La3 and La4 as the loss value.

[0088] Step 414: Optimize the weight parameters of the twin super-network module composed of the first super-network module and the corresponding second super-network module according to the loss value to obtain a candidate sub-network structure in the twin super-network module.

[0089] When calculating the loss value, the embodiment of the present application calculates the loss value for each first sampling sub-network based on the difference between the output vector of the first sampling sub-network and the mean of the output vectors of all second sampling sub-networks. That is, in the weight parameter optimization process, the embodiment of the present application forms a common goal for all paths (using sub-networks). Therefore, compared with the optimization scheme in which each first sampling sub-network calculates the loss value only based on the difference between the output vectors of the second sampling sub-network corresponding to itself, the embodiment of the present application avoids the problem of unstable optimization process and difficult convergence caused by the fact that different sampling sub-networks are optimized only based on the corresponding single sampling sub-network without a common optimization goal, thereby improving the stability of the optimization process.

[0090] Step 416: For each candidate sub-network structure, a performance evaluation is performed on the candidate sub-network structure based on the weight parameters of the twin super-network module corresponding to the candidate sub-network structure to obtain a performance score for the candidate sub-network structure.

[0091] The performance score is used to characterize the performance of the candidate sub-network structure, such as accuracy, convergence, etc. The higher the performance score, the better the performance of the candidate sub-network structure.

[0092] Furthermore, the performance score of the candidate sub-network structure can be obtained through the following steps:

[0093] Obtain verification samples and perform data enhancement processing on the verification samples to obtain enhanced verification samples;

[0094] For each candidate sub-network structure, the enhanced verification sample is input into the candidate sub-network structure so that the candidate sub-network structure obtains a candidate output vector based on the weight parameters of the corresponding twin super-network module;

[0095] Input the same enhanced verification sample into each second sampling sub-network in the second super network module corresponding to the candidate sub-network structure, and obtain the output vector of each second sampling sub-network;

[0096] Calculate the mean of the output vectors of each second sampling sub-network as the candidate mean of the entire path;

[0097] Based on the difference between the candidate output vector and the full path candidate mean, the performance score of the candidate sub-network structure is obtained.

[0098] Specifically, for example, the reciprocal of the difference between the candidate output vector and the full path candidate mean can be determined as the performance score of the candidate sub-network structure, and so on.

[0099] In the embodiment of the present application, when the performance evaluation is performed on each candidate sub-network structure, the performance score is obtained based on the difference between the candidate output vector of the candidate sub-network structure and the mean of the output vectors of all the second sampling sub-networks. That is to say, in the performance evaluation process, the embodiment of the present application forms a common goal for all paths (using sub-networks). Therefore, compared with the solution in which each candidate sub-network structure obtains a performance score based only on the difference between the output vectors of the second sampling sub-network corresponding to itself, the performance score obtained in the embodiment of the present application can better characterize the true performance status of each candidate sub-network structure. That is to say, the performance score obtained in the embodiment of the present application is more accurate. Therefore, based on the more accurate performance score, the performance of the target network structure determined is also higher. Therefore, the embodiment of the present application further improves the accuracy of the search results.

[0100] In addition, in other embodiments of the present application, for each twin super network module, a performance score can also be obtained based on the difference between the candidate output vector of the candidate sub-network structure and the mean of the output vectors of all sub-networks contained in the twin super network module. In this way, since all sub-networks are taken into account, the accuracy of the performance score can be further improved.

[0101] Step 418: Add the performance scores of the candidate sub-network structures to obtain the performance score of the candidate network structure.

[0102] Among them, the candidate network structure is composed of various candidate sub-network structures.

[0103] Step 420 , determine whether the preset optimization termination condition is met; if so, execute step 422 ; if not, adjust the weight parameters of the twin hypernetwork based on the performance score, and return to step 410 .

[0104] The optimization termination condition can be set according to actual needs. For example, the optimization termination condition can be determined to be satisfied when the number of optimizations reaches a preset number threshold; the optimization termination condition can also be determined to be satisfied when the loss value is less than a preset loss value threshold, etc. In the embodiments of the present application, the specific content of the optimization termination condition is not limited.

[0105] Step 422 : determining a target network structure from a plurality of candidate network structures according to the performance scores of the candidate network structures.

[0106] Specifically, for example, a preset number of candidate network structures may be selected as the target network structure in descending order of performance scores.

[0107] In an embodiment of the present application, after obtaining two supernetworks, the supernetworks are optimized based on the output vectors of the two supernetworks to obtain a candidate network structure. That is to say, the above optimization process is carried out by self-supervised comparison between two supernetworks with the same structure, without relying on an external network model obtained based on label training to provide supervision information. Therefore, the problem of structural deviation in search results due to reliance on supervision information provided by an external network model is avoided, thereby improving the accuracy of structural search.

[0108] At the same time, in the embodiment of the present application, in the process of performing a neural network structure search, the super network is divided into blocks, and then each module obtained by the division is searched separately. Compared with the entire search space, the number of search space layers corresponding to each module after the block is smaller, so the search amount is significantly reduced, thereby improving the efficiency of the neural network structure search.

[0109] In addition, on the one hand: when calculating the loss value, the embodiment of the present application calculates the loss value for each first sampling sub-network based on the difference between the output vector of the first sampling sub-network and the mean of the output vectors of all the second sampling sub-networks, thereby avoiding the problem of unstable optimization process and difficult convergence caused by the fact that different sampling sub-networks are optimized only based on the corresponding single sampling sub-network without a common optimization target, thereby improving the stability of the optimization process; on the other hand: when evaluating the performance, the performance score is obtained based on the difference between the candidate output vector of the candidate sub-network structure and the mean of the output vectors of all the second sampling sub-networks, so that the obtained performance score can better represent the true performance status of each candidate sub-network structure. Based on the more accurate performance score, the performance of the determined target network structure is also higher, further improving the accuracy of the search results.

[0110] The neural network structure search method of this embodiment can be executed by any appropriate electronic device with data processing capabilities, including but not limited to: a server, etc.

[0111] Example 3:

[0112] See also Figure 8 , Figure 8 This is a block diagram of a neural network structure search device according to the third embodiment of the present application. The neural network structure search device provided in the embodiment of the present application includes:

[0113] A twin supernetwork acquisition module 802 is configured to acquire a twin supernetwork constructed based on a preset search space, wherein the twin supernetwork includes a first supernetwork and a second supernetwork; the second supernetwork has the same network structure as the first supernetwork, and the network weight of the second supernetwork is obtained based on the network weight of the first supernetwork;

[0114] A candidate network structure obtaining module 804 is configured to obtain a loss value based on the output vector of the first supernetwork, the output vector of the second supernetwork, and a preset loss function, and optimize the weight parameters of the twin supernetwork based on the loss value to obtain a candidate network structure;

[0115] A performance evaluation module 806 is configured to perform a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure;

[0116] The weight parameter adjustment module 808 is used to adjust the weight parameters of the twin super network based on the performance score, and return a loss value obtained based on the output vector of the first super network, the output vector of the second super network, and a preset loss function, and optimize the weight parameters of the twin super network based on the loss value to obtain a candidate network structure until a preset optimization termination condition is met;

[0117] The target network structure determination module 810 is configured to determine a target network structure from a plurality of candidate network structures according to the performance scores of the candidate network structures.

[0118] Optionally, in some embodiments, the network weight of the second super network is obtained by performing an exponential sliding average process on the network weight of the first super network.

[0119] Optionally, in some embodiments, the candidate network structure obtaining module 804 is specifically configured to:

[0120] The first supernetwork is divided into blocks to obtain a plurality of first supernetwork modules; the second supernetwork is divided into blocks to obtain a plurality of second supernetwork modules; wherein the first supernetwork modules correspond to the second supernetwork modules one by one;

[0121] For each first super-network module, a loss value is obtained according to the output vector of the first super-network module, the output vector of the corresponding second super-network module and the preset loss function. Based on the loss value, the weight parameters of the twin super-network module composed of the first super-network module and the corresponding second super-network module are optimized to obtain a candidate sub-network structure in the twin super-network module; the candidate network structure is composed of a combination of the candidate sub-network structures.

[0122] Optionally, in some embodiments, the candidate network structure obtaining module 804, when executing the step of obtaining a loss value for each first super network module according to the output vector of the first super network module, the output vector of the corresponding second super network module, and a preset loss function, and optimizing the weight parameters of the twin super network module composed of the first super network module and the corresponding second super network module based on the loss value to obtain a candidate sub-network structure in the twin super network module, is specifically used to:

[0123] Obtain optimized samples, and perform data enhancement processing on the optimized samples to obtain enhanced samples;

[0124] Performing convolution processing on the enhanced sample to obtain a first enhanced optimized sample and a second enhanced optimized sample;

[0125] For each first super network module, input the first enhanced optimized sample into the first super network module to obtain a first output vector; input the second enhanced optimized sample into the corresponding second super network module to obtain a second output vector;

[0126] Based on the first output vector and the second output vector, a preset loss function is used to obtain a loss value;

[0127] The weight parameters of the twin super-network module composed of the first super-network module and the corresponding second super-network module are optimized according to the loss value to obtain a candidate sub-network structure in the twin super-network module.

[0128] Optionally, in some embodiments, when executing, for each first super network module, inputting the first enhanced optimized sample into the first super network module to obtain the first output vector, the candidate network structure obtaining module 804 is specifically configured to:

[0129] For each first super network module, path sampling is performed on the first super network module according to a preset sampling strategy to obtain multiple first sampling sub-networks; the first enhanced optimized sample is input into each first sampling sub-network respectively to obtain a first output vector of each first sampling sub-network;

[0130] When the candidate network structure obtaining module 804 performs the step of inputting the second enhanced optimized sample into the second super network module corresponding to the first super network module to obtain the second output vector, it is specifically used to:

[0131] According to a preset sampling strategy, path sampling is performed on the second super network module corresponding to the first super network module to obtain multiple second sampling sub-networks; the second enhanced optimized sample is input into each second sampling sub-network respectively to obtain a second output vector of each second sampling sub-network;

[0132] When executing the step of obtaining a loss value using a preset loss function based on the first output vector and the second output vector, the candidate network structure obtaining module 804 is specifically configured to:

[0133] Calculate the mean of all second output vectors as the full path mean;

[0134] For each first sampling subnetwork, subtract the first output vector of the first sampling subnetwork from the full path mean, as the difference value corresponding to the first sampling subnetwork;

[0135] A loss value is obtained based on the sum of the difference values ​​corresponding to each first sampling sub-network.

[0136] Optionally, in some embodiments, the performance evaluation module 806 is specifically configured to:

[0137] For each candidate sub-network structure, the performance of the candidate sub-network structure is evaluated based on the weight parameters of the twin super-network module corresponding to the candidate sub-network structure to obtain the performance score of the candidate sub-network structure;

[0138] The performance scores of each candidate sub-network structure are added together to obtain the performance score of the candidate network structure.

[0139] Optionally, in some embodiments, when the performance evaluation module 806 performs the step of evaluating the performance of each candidate sub-network structure based on the weight parameters of the twin super-network module corresponding to the candidate sub-network structure to obtain the performance score of the candidate sub-network structure, it is specifically configured to:

[0140] Obtain verification samples and perform data enhancement processing on the verification samples to obtain enhanced verification samples;

[0141] For each candidate sub-network structure, the enhanced verification sample is input into the candidate sub-network structure so that the candidate sub-network structure obtains a candidate output vector based on the weight parameters of the corresponding twin super-network module;

[0142] Input the enhanced verification samples into each second sampling sub-network in the second super network module corresponding to the candidate sub-network structure, and obtain the output vector of each second sampling sub-network;

[0143] Calculate the mean of the output vectors of each second sampling sub-network as the candidate mean of the entire path;

[0144] Based on the difference between the candidate output vector and the full path candidate mean, the performance score of the candidate sub-network structure is obtained.

[0145] Optionally, in some embodiments, the target network structure determination module 810 is specifically configured to:

[0146] A preset number of candidate network structures are selected as the target network structure in descending order of performance scores.

[0147] Optionally, in some embodiments, the convolution candidate operators of the preset search space include: the residual module in the ResNet network; the candidate operators of the preset search space also include: the self-attention candidate operator formed by combining the depthwise separable convolution module and the NLBlock module, and / or, the self-attention candidate operator obtained by replacing the relative position encoding branch in the BoTBlock module with the depthwise separable convolution module.

[0148] The neural network structure search device of this embodiment is used to implement the corresponding neural network structure search methods in the aforementioned multiple method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be described in detail here. In addition, the functional implementation of each module in the neural network structure search device of this embodiment can refer to the corresponding descriptions of the aforementioned method embodiments and will not be described in detail here.

[0149] Example 4:

[0150] Reference Figure 9 , shows a structural diagram of an electronic device according to the fourth embodiment of the present application. The specific embodiment of the present application does not limit the specific implementation of the electronic device.

[0151] like Figure 9 As shown, the electronic device may include: a processor (processor) 902 , a communication interface (Communications Interface) 904 , a memory (memory) 906 , and a communication bus 908 .

[0152] in:

[0153] The processor 902 , the communication interface 904 , and the memory 906 communicate with each other via a communication bus 908 .

[0154] The communication interface 904 is used to communicate with other electronic devices or servers.

[0155] Processor 902 is used to execute program 910, and specifically can execute the relevant steps in the above-mentioned neural network structure search method embodiment.

[0156] Specifically, the program 910 may include program codes, which include computer operation instructions.

[0157] The processor 902 may be a CPU, an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. The one or more processors included in the smart device may be processors of the same type, such as one or more CPUs, or processors of different types, such as one or more CPUs and one or more ASICs.

[0158] The memory 906 is used to store the program 910. The memory 906 may include a high-speed RAM memory, and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0159] Program 910 can be specifically used to enable the processor 902 to perform the following operations: obtain a twin supernetwork constructed based on a preset search space, the twin supernetwork including a first supernetwork and a second supernetwork; the second supernetwork has the same network structure as the first supernetwork, and the network weight of the second supernetwork is obtained based on the network weight of the first supernetwork; obtain a loss value based on the output vector of the first supernetwork, the output vector of the second supernetwork and a preset loss function, optimize the weight parameters of the twin supernetwork based on the loss value, and obtain a candidate network structure; perform performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure; adjust the weight parameters of the twin supernetwork based on the performance score, and return the step of obtaining a candidate network structure based on the loss value obtained according to the output vector of the first supernetwork, the output vector of the second supernetwork and the preset loss function, and optimize the weight parameters of the twin supernetwork based on the loss value until the preset optimization termination condition is met; determine the target network structure from multiple candidate network structures based on the performance scores of each candidate network structure.

[0160] The specific implementation of each step in program 910 can be found in the corresponding descriptions of the corresponding steps and units in the above-mentioned embodiment of the neural network structure search method, and will not be repeated here. Those skilled in the art will clearly understand that for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding process descriptions in the above-mentioned method embodiment, and will not be repeated here.

[0161] Through the electronic device of this embodiment, after obtaining two hypernetworks, the hypernetworks are optimized based on the output vectors of the two hypernetworks to obtain a candidate network structure. That is to say, the above optimization process is carried out through a self-supervised comparison between two hypernetworks with the same structure, without relying on an external network model obtained through label training to provide supervisory information. Therefore, the problem of structural deviation in search results due to reliance on supervisory information provided by an external network model is avoided, thereby improving the accuracy of structural search.

[0162] An embodiment of the present application also provides a computer program product, including computer instructions, which instruct a computing device to perform operations corresponding to any neural network structure search method in the above-mentioned multiple method embodiments.

[0163] It should be pointed out that, according to the needs of implementation, the various components / steps described in the embodiments of the present application can be split into more components / steps, or two or more components / steps or partial operations of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of the present application.

[0164] The methods according to the embodiments of the present application described above can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code originally stored in a remote recording medium or non-transitory machine-readable medium downloaded via a network and then stored in a local recording medium. Thus, the methods described herein can be stored in such software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It will be understood that a computer, processor, microprocessor controller, or programmable hardware includes a storage component (e.g., RAM, ROM, flash memory, etc.) that can store or receive software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the neural network structure search method described herein is implemented. Furthermore, when a general-purpose computer accesses the code for implementing the neural network structure search method shown herein, the execution of the code converts the general-purpose computer into a dedicated computer for executing the neural network structure search method shown herein.

[0165] Those skilled in the art will appreciate that the units and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of this application.

[0166] The above implementation methods are only used to illustrate the embodiments of the present application, and are not intended to limit the embodiments of the present application. Ordinary technicians in the relevant technical field can make various changes and modifications without departing from the spirit and scope of the embodiments of the present application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of the present application, and the scope of patent protection of the embodiments of the present application should be defined by the claims.

Claims

1. A neural network structure search method, comprising: Acquire a twin supernetwork constructed based on a preset search space, wherein the twin supernetwork includes a first supernetwork and a second supernetwork; The second supernetwork has the same network structure as the first supernetwork, and the network weight of the second supernetwork is obtained based on the network weight of the first supernetwork; A loss value is obtained according to the output vector of the first supernetwork, the output vector of the second supernetwork, and a preset loss function, and a weight parameter of the twin supernetwork is optimized based on the loss value to obtain a candidate network structure; the output vector of the first supernetwork and the output vector of the second supernetwork are obtained based on an optimized sample, and the optimized sample is image data; Performing a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure; Based on the performance score, the weight parameters of the twin super network are adjusted, and the loss value obtained according to the output vector of the first super network, the output vector of the second super network and the preset loss function is returned, and the weight parameters of the twin super network are optimized based on the loss value to obtain a candidate network structure until a preset optimization termination condition is met; According to the performance scores of the candidate network structures, the target network structure is determined from multiple candidate network structures.

2. The method according to claim 1, wherein The network weight of the second super network is obtained by performing exponential sliding average processing on the network weight of the first super network.

3. The method according to claim 2, wherein: The step of obtaining a loss value according to the output vector of the first supernetwork, the output vector of the second supernetwork, and a preset loss function, and optimizing the weight parameters of the twin supernetwork based on the loss value to obtain a candidate network structure includes: The first supernetwork is divided into blocks to obtain a plurality of first supernetwork modules; the second supernetwork is divided into blocks to obtain a plurality of second supernetwork modules; wherein the first supernetwork modules correspond to the second supernetwork modules in a one-to-one manner; For each first super-network module, a loss value is obtained according to the output vector of the first super-network module, the output vector of the corresponding second super-network module and the preset loss function. Based on the loss value, the weight parameters of the twin super-network module composed of the first super-network module and the corresponding second super-network module are optimized to obtain a candidate sub-network structure in the twin super-network module; the candidate network structure is composed of a combination of the candidate sub-network structures.

4. The method according to claim 3, wherein: For each first super-network module, a loss value is obtained according to the output vector of the first super-network module, the output vector of the corresponding second super-network module, and a preset loss function; based on the loss value, the weight parameters of the twin super-network module composed of the first super-network module and the corresponding second super-network module are optimized to obtain a candidate sub-network structure in the twin super-network module, including: Obtaining an optimized sample, and performing data enhancement processing on the optimized sample to obtain an enhanced sample; Performing convolution processing on the enhanced sample to obtain a first enhanced optimized sample and a second enhanced optimized sample; For each first super network module, input the first enhanced optimized sample into the first super network module to obtain a first output vector; input the second enhanced optimized sample into the corresponding second super network module to obtain a second output vector; Obtaining a loss value using a preset loss function based on the first output vector and the second output vector; The weight parameters of the twin super-network module composed of the first super-network module and the corresponding second super-network module are optimized according to the loss value to obtain a candidate sub-network structure in the twin super-network module.

5. The method according to claim 4, wherein The step of inputting the first enhanced optimized sample into each first super-network module to obtain a first output vector includes: For each first supernetwork module, path sampling is performed on the first supernetwork module according to a preset sampling strategy to obtain multiple first sampling subnetworks; the first enhanced optimized sample is input into each first sampling subnetwork respectively to obtain a first output vector of each first sampling subnetwork; Inputting the second enhanced optimized sample into a second super-network module corresponding to the first super-network module to obtain a second output vector includes: Performing path sampling on the second supernetwork module corresponding to the first supernetwork module according to the preset sampling strategy to obtain a plurality of second sampling subnetworks; inputting the second enhanced optimized sample into each of the second sampling subnetworks to obtain a second output vector of each of the second sampling subnetworks; The obtaining a loss value by using a preset loss function based on the first output vector and the second output vector includes: Calculate the mean of all second output vectors as the full path mean; For each first sampling subnetwork, subtract the first output vector of the first sampling subnetwork from the full path mean, as the difference value corresponding to the first sampling subnetwork; A loss value is obtained based on the sum of the difference values ​​corresponding to each first sampling sub-network.

6. The method according to claim 5, wherein: The performing performance evaluation on the candidate network structure based on the weight parameter to obtain a performance score of the candidate network structure includes: For each candidate sub-network structure, the performance of the candidate sub-network structure is evaluated based on the weight parameters of the twin super-network module corresponding to the candidate sub-network structure to obtain the performance score of the candidate sub-network structure; The performance scores of the candidate sub-network structures are added together to obtain the performance score of the candidate network structure.

7. The method according to claim 6, wherein: For each candidate sub-network structure, the performance of the candidate sub-network structure is evaluated based on the weight parameters of the twin super-network module corresponding to the candidate sub-network structure to obtain a performance score of the candidate sub-network structure, including: Obtaining a verification sample, and performing data enhancement processing on the verification sample to obtain an enhanced verification sample; For each candidate sub-network structure, input the enhanced verification sample into the candidate sub-network structure so that the candidate sub-network structure obtains a candidate output vector based on the weight parameters of the corresponding twin super-network module; Inputting the enhanced verification samples into each second sampling sub-network in the second super network module corresponding to the candidate sub-network structure, respectively, to obtain an output vector of each second sampling sub-network; Calculating the mean of the output vectors of each second sampling sub-network as the candidate mean of the entire path; Based on the difference between the candidate output vector and the full path candidate mean, a performance score of the candidate sub-network structure is obtained.

8. The method according to any one of claims 1 to 7, wherein: The step of determining a target network structure from a plurality of candidate network structures according to the performance scores of the candidate network structures includes: A preset number of candidate network structures are selected as the target network structure in descending order of performance scores.

9. The method according to any one of claims 1 to 7, wherein: The convolution candidate operators of the preset search space include: the residual module in the ResNet network; the candidate operators of the preset search space also include: the self-attention candidate operator composed of a depth-separable convolution module and an NLBlock module, and / or the self-attention candidate operator obtained by replacing the relative position encoding branch in the BoTBlock module with the depth-separable convolution module.

10. A neural network structure search device, comprising: A twin supernetwork acquisition module, configured to acquire a twin supernetwork constructed based on a preset search space, wherein the twin supernetwork includes a first supernetwork and a second supernetwork; The second supernetwork has the same network structure as the first supernetwork, and the network weight of the second supernetwork is obtained based on the network weight of the first supernetwork; A candidate network structure obtaining module is used to obtain a loss value based on the output vector of the first super network, the output vector of the second super network, and a preset loss function, and optimize the weight parameters of the twin super network based on the loss value to obtain a candidate network structure; the output vector of the first super network and the output vector of the second super network are obtained based on an optimized sample, and the optimized sample is image data; A performance evaluation module, configured to perform a performance evaluation on the candidate network structure based on the weight parameters to obtain a performance score of the candidate network structure; A weight parameter adjustment module is used to adjust the weight parameters of the twin super network based on the performance score, and return the loss value obtained according to the output vector of the first super network, the output vector of the second super network and the preset loss function, and optimize the weight parameters of the twin super network based on the loss value to obtain a candidate network structure until a preset optimization termination condition is met; The target network structure determination module is used to determine the target network structure from multiple candidate network structures based on the performance score of each candidate network structure.

11. An electronic device comprising: A processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; The memory is used to store at least one executable instruction, and the executable instruction enables the processor to perform operations corresponding to the neural network structure search method according to any one of claims 1 to 9.

12. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the neural network structure search method as described in any one of claims 1 to 9.

13. A computer program product comprising computer instructions, wherein the computer instructions instruct a computing device to perform operations corresponding to the neural network structure search method according to any one of claims 1 to 9.

Images