A cross-domain neural evaluation method, electronic device and readable storage medium

Abstract

The present invention discloses a cross-domain neural evaluation method, electronic device and readable storage medium, which relates to the field of neural architecture search. The method includes the steps of: constructing a coding source architecture space, an assistant teaching architecture space and a target architecture space; training a cross-domain neural evaluator through the coding source architecture space, the assistant teaching architecture space and the target architecture space; using the neural architecture in the target architecture space as the input of the trained cross-domain neural evaluator, evaluating the performance values ​​of all neural architectures in the target architecture space, and completing the cross-domain neural evaluation. The method uses the assistant teaching architecture space as a bridge between the source architecture space and the target architecture space, so that the migration of the source architecture space to the target space is smoother, and through the subspace division method, the cross-domain neural evaluation is finally realized, and the evaluation is more accurate.

Description

Technical Field

[0001] The present invention relates to the field of neural architecture search, and in particular to a cross-domain neural evaluation method, electronic device and readable storage medium. Background Art

[0002] Neural architecture search (NAS) can automatically design promising architectures for practical deep neural networks for the task at hand, such as image classification, language modeling, and medical image segmentation. In recent years, many architectures searched by NAS methods have surpassed the performance of manually designed networks. Typically, executing NAS methods usually requires hundreds of GPUs to run for a week or even a month, all of which is spent on performance evaluation of neural architectures. Unfortunately, such a large amount of computing resources is not available to most interested researchers and companies. Therefore, NAS researchers are increasingly focusing on improving the search rate of NAS by developing accelerated evaluation methods so that the search can be completed within an acceptable search cost.

[0003] In general, most of the existing acceleration methods are designed based on the following strategies: reducing the search space, searching a small proxy dataset, using Monte Carlo to search trees in promising areas, and sharing weights (i.e., weight sharing). Among them, weight sharing is the most widely used acceleration method in the NAS community. It pre-trains a manually designed super network and then searches for sub-networks in the super network. Weights of the same component can be shared between different sub-networks and trained super networks without further training, which greatly improves the efficiency of NAS. With the help of weight sharing, many NAS methods reduce their computational overhead from thousands of GPU Days to a few GPU Days, such as DARTS. Even less than one GPU Day, for example, PC-DARTS. However, weight sharing also suffers from unstable performance and strong performance collapse, which severely limits its application.

[0004] Neural evaluators are another type of acceleration method that can directly estimate the performance of neural architectures without any training. This is a promising research direction, and researchers have conducted a lot of related research in this regard. For example, E2EPP uses random forests as neural evaluators and greatly reduces the time required in NAS. SSANA uses graph convolutional networks (GCNs) as neural evaluators and makes full use of unlabeled architectures to improve evaluation accuracy. In general, neural evaluators need to first sample enough architectures in the target search space, that is, sample in the search space of the NAS method, and then fully train these architectures to obtain their performance. Finally, the architecture and its performance form a training data set to construct an evaluator. In practice, at least hundreds of annotated architectures need to be sampled to ensure a relatively accurate evaluation. Therefore, neural evaluators still require a lot of computational costs, which mainly come from the need to sample architectures and train them.

[0005] There are many benchmark datasets for NAS research, such as NAS-Bench-101, NAS-Bench-201, NAS-Bench-nlp, NAS-Bench-ASR, etc. These datasets have a large number of actual performance of architectures trained from scratch, and have the potential to build neural evaluators without the need for additional labeled data. However, the search space of these datasets is relatively small and different from the currently popular search space, that is, the search space of the most advanced NAS algorithms, such as DARTS and ProxylessNAS. Therefore, these trained benchmark architectures cannot be directly used to build neural evaluators, let alone be applied to the evaluation of cross-domain neural architectures. Summary of the invention

[0006] In view of the above-mentioned deficiencies in the prior art, the present invention provides a cross-domain neural evaluation method, electronic device and readable storage medium to solve the problem that the existing architecture cannot be applied to the evaluation of cross-domain neural architecture.

[0007] In order to achieve the above-mentioned object of the invention, the technical solution adopted by the present invention is:

[0008] A cross-domain neural evaluation method is provided, which comprises the following steps:

[0009] S1. Determine the target domain, construct the source architecture space, the assistant architecture space and the target architecture space respectively, record the number of generations as 1, and initialize the feature extraction network and the regression hypothesis model;

[0010] The source architecture space, the assistant architecture space, and the target architecture space all contain several neural architectures; the neural architectures in the source architecture space are neural architectures with known performance values ​​used in the target domain; the neural architectures in the assistant architecture space and the target architecture space are neural architectures used in other domains;

[0011] S2, uniformly encode the neural architectures in the source architecture space, the assistant architecture space, and the target architecture space to obtain the encoded source architecture, the encoded assistant architecture, and the encoded target architecture respectively;

[0012] S3, determine whether the current number of generations is less than or equal to the first set value, if so, proceed to step S4; otherwise, proceed to step S11;

[0013] S4, respectively obtaining the features of the encoded source architecture and the features of the encoded teaching assistant architecture through the current feature extraction network; that is, obtaining the source architecture features and the teaching assistant architecture features respectively;

[0014] S5. Based on the source architecture features and the teaching assistant architecture features, the performance values ​​of each architecture in the source architecture space and the teaching assistant architecture space are evaluated respectively through the current regression hypothesis model to obtain the performance evaluation values ​​of each architecture in the source architecture space and the teaching assistant architecture space;

[0015] S6. Update the number of categories of neural architectures in the source architecture space and the assistant architecture space according to the current number of generations; divide the source architecture space and the assistant architecture space into several subspaces according to the number of categories and the performance evaluation value; the number of categories in the subspace where the neural architecture is located is the category value of the neural architecture;

[0016] S7, obtaining the weight of the corresponding architecture feature based on the category value of the neural architecture in the subspace;

[0017] S8, performing LMMD calculation on the source architecture features, the teaching assistant architecture features, and the weights of the architecture features to obtain an LMMD calculation value;

[0018] S9, constructing an objective function according to the performance value of the architecture feature and the corresponding LMMD calculation value, and updating the parameters in the feature extraction network and the regression hypothesis model with the purpose of minimizing the objective function;

[0019] S10, add 1 to the current generation number, and return to step S3;

[0020] S11, replacing the encoded teaching assistant architecture with the encoded target architecture, replacing the teaching assistant architecture space with the target architecture space, and updating the parameters in the current feature extraction network and the regression hypothesis model using the same method as steps S3 to S10 until the current number of generations reaches a second set value, and using the current feature extraction network and the regression hypothesis model as the trained cross-domain neural evaluator;

[0021] S12. Use the neural architecture in the target architecture space as the input of the trained cross-domain neural evaluator to evaluate the performance values ​​of all neural architectures in the target architecture space in the target domain, and complete the cross-domain neural evaluation.

[0022] Furthermore, the target areas in step S1 include: image classification, image recognition, sentiment analysis, semantic analysis, text classification, speech recognition and machine translation;

[0023] The specific methods for constructing the source architecture space, the assistant architecture space and the target architecture space in step S1 are as follows:

[0024] The purpose of designing a neural architecture is obtained according to the target domain, and a neural architecture of the same type with known performance values ​​used to achieve the purpose is selected to obtain a source architecture space;

[0025] Take DARTS space as the target architecture space;

[0026] The number of intermediate nodes of the architecture in the DARTS space is adjusted from 4 to 3 to obtain the teaching assistant architecture space.

[0027] Furthermore, the specific method for performing unified encoding in step S2 includes the following sub-steps:

[0028] S2-1. Unify the number of input operations in different neural architectures: adjust the neural architecture with 2 or more input operations into multiple single-input neural architectures by removing input operations;

[0029] S2-2, obtaining the topological structure of the single-input neural architecture obtained by adjustment;

[0030] S2-3, obtain the corresponding template adjacency matrix through the topological structure of the single input neural architecture;

[0031] S2-4, pruning the template adjacency matrix through skip connection and zeroing operation to obtain the preprocessed adjacency matrix;

[0032] S2-5, filling the penultimate row and penultimate column of the preprocessed adjacency matrix with zeros so that all adjacency matrices after zero filling have the same size, and obtaining an adjacency matrix corresponding to each neural architecture;

[0033] S2-6, use one-hot encoding to map and encode input, output, 1×1 convolution, 3×3 convolution, maximum pooling, average pooling and zeroing operations; use all-zero encoding for the remaining operation types; obtain the operation encoding matrix corresponding to each neural architecture;

[0034] Among them, the order in the operation encoding matrix corresponding to the same neural architecture corresponds to the order of each row and column of the adjacency matrix corresponding to the neural architecture; the operation encoding matrix and the adjacency matrix constitute the encoded architecture space.

[0035] Furthermore, in step S4, the specific method of obtaining the features of the encoded architecture through the feature extraction network is:

[0036] A graph convolutional neural network is established, and a fully connected layer is connected to the output of the graph convolutional neural network; the encoded architecture is used as the input of the graph convolutional neural network, and the output of the fully connected layer is used as the feature of the encoded architecture.

[0037] Furthermore, the specific method of step S6 includes the following sub-steps:

[0038] S6-1, according to the formula:

[0039]

[0040] Update the number of categories C of neural architectures in the architecture space e ; K is the maximum value of the set number of categories; cos(·) is the cosine function; π is 180°; E is the maximum value of the set generation, that is, the second set value; Indicates round down operation; e is the current generation number;

[0041] S6-2, sorting the neural architectures in the architecture space from small to large according to their performance values, to obtain a sorted architecture space;

[0042] S6-3. Divide the sorted architecture space into C e subspaces, and set the number of categories of neural architectures in the cth subspace to c; where c = 1, 2, ..., C e .

[0043] Furthermore, the specific method of step S7 includes the following sub-steps:

[0044] S7-1. Initialize a vector with the length of the current number of categories for the label of each neural architecture in the subspace;

[0045] S7-2, set the cth element in the vector corresponding to the neural architecture with c categories to 1, and set the remaining elements in the vector to 0, to obtain the label vector corresponding to the i-th neural architecture in the source architecture space Get the label vector corresponding to the jth neural architecture in the TA architecture space

[0046] S7-3, according to the formula:

[0047]

[0048]

[0049] Get the weights of the features corresponding to the i-th neural architecture in the source architecture space respectively And the weight of the feature corresponding to the jth neural architecture in the teaching assistant architecture space in Represents the value of the cth element in the label vector corresponding to the i-th neural architecture in the source architecture space; n s represents the total number of neural architectures in the source architecture space; represents the value of the cth element in the label vector corresponding to the jth neural architecture in the teaching assistant architecture space; n t is the total number of neural architectures in the TA architecture space.

[0050] Furthermore, the specific method of step S8 is:

[0051] According to the formula:

[0052]

[0053] Get LMMD calculation value Where C represents the upper limit of the number of subspaces; represents the weight of the feature corresponding to the jth neural architecture in the source architecture space; represents the weight of the feature corresponding to the i-th neural architecture in the TA architecture space; k(·) is the kernel function; is the i-th element in the source architecture feature, that is, the feature corresponding to the i-th neural architecture in the source architecture space, that is, the activation output of the feature extraction network when the i-th neural architecture in the source architecture space is input; is the jth element in the source architecture feature, that is, the feature corresponding to the jth neural architecture in the source architecture space, that is, the activation output of the feature extraction network when the jth neural architecture in the source architecture space is input; is the jth element in the teaching assistant architecture feature, that is, the feature corresponding to the jth neural architecture in the teaching assistant architecture space, that is, the activation output of the feature extraction network when the jth neural architecture in the teaching assistant architecture space is input; The i-th element in the teaching assistant architecture feature is the feature corresponding to the i-th neural architecture in the teaching assistant architecture space, that is, the activation output of the feature extraction network when the i-th neural architecture in the teaching assistant architecture space is input.

[0054] Furthermore, the expression of the objective function in step S9 is:

[0055]

[0056] Where n srepresents the total number of neural architectures in the source architecture space; ζ(·) is the loss function; W represents the parameters in the feature extraction network and the regression hypothesis model; represents the nth neural architecture in the source architecture space; Indicates that the current parameter is W and the input is The performance evaluation value when ; λ is a constant greater than 0; Calculate the value for LMMD; is the true performance value of the nth neural architecture in the source architecture space.

[0057] An electronic device is provided, the device comprising:

[0058] a memory storing executable instructions; and

[0059] The processor is configured to execute the executable instructions in the memory to implement a cross-domain neurological assessment method.

[0060] A readable storage medium is provided, on which executable instructions are stored, and when the executable instructions are executed by a processor, a cross-domain neural evaluation method is implemented.

[0061] The beneficial effects of the present invention are as follows: the present invention adopts the teaching assistant architecture space as a bridge between the source architecture space and the target architecture space, so that the migration from the source architecture space to the target space is smoother, and through the subspace division method, cross-domain neural evaluation is finally realized, and the evaluation is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 is a schematic diagram of the process of this method;

[0063] Figure 2 are examples of units in the three search spaces;

[0064] Figure 3 The performance of the evaluator under different fixed categories C in LMMD. DETAILED DESCRIPTION

[0065] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0066] like Figure 1 As shown, the cross-domain neural evaluation method includes the following steps:

[0067] S1. Determine the target domain, construct the source architecture space, the assistant architecture space and the target architecture space respectively, record the number of generations as 1, and initialize the feature extraction network and the regression hypothesis model;

[0068] The source architecture space, the assistant architecture space, and the target architecture space all contain several neural architectures; the neural architectures in the source architecture space are neural architectures with known performance values ​​used in the target domain; the neural architectures in the assistant architecture space and the target architecture space are neural architectures used in other domains;

[0069] S2, uniformly encode the neural architectures in the source architecture space, the assistant architecture space, and the target architecture space to obtain the encoded source architecture, the encoded assistant architecture, and the encoded target architecture respectively;

[0070] S3, determine whether the current generation number is less than or equal to the first set value, if so, proceed to step S4; otherwise, proceed to step S11;

[0071] S4, respectively obtaining the features of the encoded source architecture and the features of the encoded teaching assistant architecture through the current feature extraction network; that is, obtaining the source architecture features and the teaching assistant architecture features respectively;

[0072] S5. Based on the source architecture features and the teaching assistant architecture features, the performance values ​​of each architecture in the source architecture space and the teaching assistant architecture space are evaluated respectively through the current regression hypothesis model to obtain the performance evaluation values ​​of each architecture in the source architecture space and the teaching assistant architecture space;

[0073] S6. Update the number of categories of neural architectures in the source architecture space and the assistant architecture space according to the current number of generations; divide the source architecture space and the assistant architecture space into several subspaces according to the number of categories and the performance evaluation value; the number of categories in the subspace where the neural architecture is located is the category value of the neural architecture;

[0074] S7, obtaining the weight of the corresponding architecture feature based on the category value of the neural architecture in the subspace;

[0075] S8, performing LMMD calculation on the source architecture features, the teaching assistant architecture features, and the weights of the architecture features to obtain an LMMD calculation value;

[0076] S9, constructing an objective function according to the performance value of the architecture feature and the corresponding LMMD calculation value, and updating the parameters in the feature extraction network and the regression hypothesis model with the purpose of minimizing the objective function;

[0077] S10, add 1 to the current generation number, and return to step S3;

[0078] S11, replacing the encoded teaching assistant architecture with the encoded target architecture, replacing the teaching assistant architecture space with the target architecture space, and updating the parameters in the current feature extraction network and the regression hypothesis model using the same method as steps S3 to S10 until the current number of generations reaches a second set value, and using the current feature extraction network and the regression hypothesis model as the trained cross-domain neural evaluator;

[0079] S12. Use the neural architecture in the target architecture space as the input of the trained cross-domain neural evaluator to evaluate the performance values ​​of all neural architectures in the target architecture space in the target domain, and complete the cross-domain neural evaluation.

[0080] The electronic device comprises:

[0081] a memory storing executable instructions; and

[0082] The processor is configured to execute the executable instructions in the memory to implement the cross-domain neural evaluation method.

[0083] This readable storage medium stores executable instructions, which can implement a cross-domain neural evaluation method when executed by a processor.

[0084] The target fields in step S1 include: image classification, image recognition, sentiment analysis, semantic analysis, text classification, speech recognition and machine translation, etc. It should be noted that the target fields can also include other fields not listed, because neural networks have a wide range of applications. As a general method of neural networks, this method cannot exhaustively list all applicable fields. The specific methods for constructing the source architecture space, the assistant architecture space and the target architecture space in step S1 are:

[0085] The purpose of designing a neural architecture is obtained according to the target domain, and a neural architecture of the same type with known performance values ​​used to achieve the purpose is selected to obtain a source architecture space;

[0086] Take DARTS space as the target architecture space;

[0087] The number of intermediate nodes of the architecture in the DARTS space is adjusted from 4 to 3 to obtain the teaching assistant architecture space.

[0088] like Figure 2 As shown, Figure 2 (a) is a unit in NAS-Bench-101; Figure 2 (b) is a unit in NAS-Bench-201; Figure 2(c) is a unit in DARTS. "In" represents the input node, and "Out" represents the output node. Different operations are represented by 1×1, 3×3, etc. 1×1 and 3×3 represent 1×1 and 3×3 convolution operations, respectively. MP and AP are the abbreviations of maximum pooling operation and average pooling operation. In practice, it is necessary to unify the encoding strategies of different neural architecture spaces to realize cross-domain evaluators (cross-domain evaluation methods). Taking NAS-Bench-101, NAS-Bench-201, and DARTS as examples, although these three search spaces all use cells to form architectures, the representation methods of architectures in different search spaces are very different. In addition, the original encoding strategies of architectures in different search spaces are quite different, and subspace division cannot be performed directly. In order to solve this problem, the specific method for unified encoding in step S2 includes the following sub-steps:

[0089] S2-1. Unify the number of input operations in different neural architectures: adjust neural architectures with 2 or more input operations into multiple single-input architectures by deleting input operations; for example, when an architecture has 2 inputs, adjust it into 2 single-input neural architectures by retaining only one input and deleting the other inputs;

[0090] S2-2, obtaining the topological structure of the single-input neural architecture obtained by adjustment;

[0091] S2-3, obtain the corresponding template adjacency matrix through the topological structure of the single input neural architecture;

[0092] S2-4, pruning the template adjacency matrix through skip connection and zeroing operation to obtain the preprocessed adjacency matrix;

[0093] S2-5, filling the penultimate row and penultimate column of the preprocessed adjacency matrix with zeros so that all adjacency matrices after zero filling have the same size, and obtaining an adjacency matrix corresponding to each neural architecture;

[0094] S2-6, use one-hot encoding to map and encode input, output, 1×1 convolution, 3×3 convolution, maximum pooling, average pooling and zeroing operations; use all-zero encoding for the remaining operation types; obtain the operation encoding matrix corresponding to each neural architecture;

[0095] Among them, the order in the operation encoding matrix corresponding to the same neural architecture corresponds to the order of each row and column of the adjacency matrix corresponding to the architecture; the operation encoding matrix and the adjacency matrix constitute the encoded architecture space.

[0096] In step S3, the first set value is one tenth of the second set value.

[0097] The specific method for obtaining the features of the encoded architecture through the feature extraction network in step S4 is: establish a graph convolutional neural network, and connect a fully connected layer to the output end of the graph convolutional neural network; use the encoded architecture as the input of the graph convolutional neural network, and use the output of the fully connected layer as the features of the encoded architecture. The operation encoding matrix and the adjacency matrix corresponding to the same neural architecture together constitute the encoded architecture, that is, an encoded architecture space includes several encoded architectures. The architecture space is a general term for the source architecture space, the assistant architecture space, and the target architecture space, that is, the operations performed on the architecture space are applicable to the source architecture space, the assistant architecture space, and the target architecture space.

[0098] The specific method of evaluating the performance value of each architecture in the architecture space through the regression hypothesis model in step S5 is: evaluating the performance value of each architecture in the architecture space through a fully connected layer. For the neural architecture adjusted in step S2-1, the neural architecture before adjustment is regarded as the original neural architecture, and the neural architecture after adjustment is regarded as the sub-neural architecture. Since the sub-neural architecture is input into the cross-domain neural evaluator, the performance evaluation value of the sub-neural architecture is finally obtained. At this time, the performance evaluation values ​​of the sub-neural architectures belonging to the same original neural architecture need to be weighted to obtain the performance evaluation value of the original neural architecture.

[0099] The specific method of step S6 includes the following sub-steps:

[0100] S6-1, according to the formula:

[0101]

[0102] Update the number of categories C of neural architectures in the architecture space e ; K is the maximum value of the set number of categories; cos(·) is the cosine function; π is 180°; E is the maximum value of the set generation, that is, the second set value; Indicates round down operation; e is the current generation number;

[0103] S6-2, sorting the neural architectures in the architecture space from small to large according to their performance values, to obtain a sorted architecture space;

[0104] S6-3. Divide the sorted architecture space into C e subspaces, and set the number of categories of neural architectures in the cth subspace to c; where c = 1, 2, ..., C e .

[0105] The specific method of step S7 includes the following sub-steps:

[0106] S7-1. Initialize a vector with the length of the current number of categories for the label of each neural architecture in the subspace;

[0107] S7-2, set the cth element in the vector corresponding to the neural architecture with c categories to 1, and set the remaining elements in the vector to 0, to obtain the label vector corresponding to the i-th neural architecture in the source architecture space Get the label vector corresponding to the jth neural architecture in the TA architecture space

[0108] S7-3, according to the formula:

[0109]

[0110]

[0111] Get the weights of the features corresponding to the i-th neural architecture in the source architecture space respectively And the weight of the feature corresponding to the jth neural architecture in the teaching assistant architecture space in Represents the value of the cth element in the label vector corresponding to the i-th neural architecture in the source architecture space; n s Indicates the total number of architectures in the source architecture space; represents the value of the cth element in the label vector corresponding to the jth neural architecture in the teaching assistant architecture space; n t is the total number of architectures in the TA architecture space.

[0112] The specific method of step S8 is: according to the formula:

[0113]

[0114] Get LMMD calculation value Where C represents the upper limit of the number of subspaces; represents the weight of the feature corresponding to the jth neural architecture in the source architecture space; represents the weight of the feature corresponding to the i-th neural architecture in the TA architecture space; k(·) is the kernel function; is the i-th element in the source architecture feature, that is, the feature corresponding to the i-th neural architecture in the source architecture space, that is, the activation output of the feature extraction network when the i-th neural architecture in the source architecture space is input; is the jth element in the source architecture feature, that is, the feature corresponding to the jth neural architecture in the source architecture space, that is, the activation output of the feature extraction network when the jth neural architecture in the source architecture space is input; is the jth element in the teaching assistant architecture feature, that is, the feature corresponding to the jth neural architecture in the teaching assistant architecture space, that is, the activation output of the feature extraction network when the jth neural architecture in the teaching assistant architecture space is input; The i-th element in the teaching assistant architecture feature is the feature corresponding to the i-th neural architecture in the teaching assistant architecture space, that is, the activation output of the feature extraction network when the i-th neural architecture in the teaching assistant architecture space is input.

[0115] The expression of the objective function in step S9 is:

[0116]

[0117] Where n s represents the total number of neural architectures in the source architecture space; ζ(·) is the loss function, and the mean square error (MSE) is preferred; W represents the parameters in the feature extraction network and the regression hypothesis model; represents the nth neural architecture in the source architecture space; P(W, ) indicates that the current parameter is W and the input is The performance evaluation value when ; λ is a constant greater than 0; Calculate the value for LMMD; is the true performance value of the nth neural architecture in the source architecture space.

[0118] In the specific implementation process, steps S3 to S10 are the process of training the feature extraction network and regression hypothesis model using the source architecture space and the teaching assistant architecture space. The encoded teaching assistant architecture is replaced with the encoded target architecture, and the teaching assistant architecture space is replaced with the target architecture space. The parameters in the current feature extraction network and regression hypothesis model are updated using the same method as steps S3 to S10, that is, the source architecture space and the target architecture space are used to train the feature extraction network and the regression hypothesis model. In this process, only the data corresponding to the teaching assistant architecture space needs to be replaced with the data corresponding to the target architecture space, and the specific calculation formula remains unchanged. Specifically, the same method as step S4 needs to be used to obtain the source architecture features and the target architecture features; the same method as step S5 needs to be used to obtain the performance evaluation values ​​of each neural architecture in the source architecture space and the target architecture space; the same method as step S6 needs to be used The method updates the number of categories of neural architectures in the source architecture space and the target architecture space according to the current number of generations; divides the source architecture space and the target architecture space into several subspaces according to the number of categories and the performance evaluation values; the same method as step S8 is required to perform LMMD calculation on the source architecture features, the target architecture features and the weights of the architecture features to obtain LMMD calculation values; the same method as step S9 is required to construct an objective function according to the performance value of the architecture features and the corresponding LMMD calculation value, and update the parameters in the feature extraction network and the regression hypothesis model with the purpose of minimizing the objective function; until the current number of generations reaches the second set value, the current feature extraction network and the regression hypothesis model are used as the trained cross-domain neural evaluator.

[0119] During the evaluation process of step S12, it is only necessary to input the neural architecture in the target architecture space into the trained cross-domain neural evaluator, and the trained cross-domain neural evaluator can output the performance evaluation value of the corresponding neural architecture.

[0120] In the specific implementation process, taking the target field as the field of image classification as an example, the purpose of designing the neural architecture is to perform image classification. The existing neural network models with known performance values ​​for image classification include LeNet, AlexNet, VGG, ResNet, etc. These mature neural network models include several neural architectures, and the data sets formed by these neural architectures are used as the source architecture space. The data sets formed by the neural architectures of neural network models for non-image classification such as sentiment analysis, semantic analysis, speech recognition and machine translation are used as the target architecture space. Through the method described in the present invention, the performance values ​​of the neural architectures originally used in sentiment analysis, semantic analysis, speech recognition or machine translation used in image classification can be obtained, thereby completing cross-domain neural evaluation.

[0121] In one embodiment of the present invention, NAS-Bench-101 and NAS-Bench-201 are selected as the source architecture space and DARTS is selected as the target architecture space. In addition, the training labels in NAS-Bench-101 and NAS-Bench-201 will be normalized respectively. After normalization, the distribution of performance values ​​in the two datasets will be more similar. It can alleviate the problem caused by the fact that the performance values ​​in NAS-Bench-101 are generally higher than those in NAS-Bench-201. We set the training batch size to 1000 and the learning rate to 2×10 -3 In addition, since the neural architectures in multiple spaces are more complicated, we set the number of hidden layers to 5.

[0122] The total number of training epochs is set to 350, and the cosine learning rate decay method is adopted, and the decay rate is set to 4×10 -5 . Regularization methods such as labeling smoothing and AutoAugment are used. In addition, a squeeze-and-excitation module is connected after each cell. Finally, the training batch size on ImageNet is set to 512 and the learning rate is 0.2. As for the training details of CIFAR-10, we follow the previous NAS method training conventions.

[0123] Table 1: Comparison with state-of-the-art neural critic-assisted NAS methods on ImageNet

[0124]

[0125] As can be seen from Table 1, the state-of-the-art neural critic-assisted NAS methods are all from recent years. In particular, we report some recent neural critic-assisted NAS methods (NAO, NeuralPredictor, and SemiNAS) in the second part of Table 1. As can be seen from Table 1, the number of parameters and FLOPs (floating point operations, which is used to measure the complexity of the model) of the neural architectures searched by all reported methods are similar (under mobile device settings). Our method (CDP) takes only 0.1 GPU Days to complete the evaluation, which is one of the lowest cost evaluation methods. Specifically, compared with other neural critic-assisted NAS methods, the search cost of CDP is 2000 times less than NAO and 40 times less than SemiNAS. CDP makes full use of the large amount of existing data without training neural architectures sampled in the search space, which is the main reason why the cost of CDP is so low. In addition, the cost of CDP is also competitive with PC-DARTS, which is a state-of-the-art NAS method using weight sharing. Despite the low evaluation cost, CDP achieves the best top-1 accuracy among all compared algorithms.

[0126] Table 2: Comparison with current state-of-the-art NAS methods on CIFAR-10

[0127]

[0128] As shown in Table 2, in addition to the methods compared in Table 1, we also compare with three other state-of-the-art methods on this dataset. They are ENAS, P-DARTS, and PVLL-NAS. The comparison results on CIFAR-10 are shown in Table 2. The number of parameters of all compared methods is similar. In terms of evaluation cost, CDP only requires 0.1 GPU Days, which is one of the lowest cost methods in Table 2. Although CDP has the lowest cost, the architecture searched by CDP still outperforms all NAS methods compared in Table 2.

[0129] Since this method is a cross-domain neural evaluation method, no experiments can be performed on NAS-Bench-101 and NAS-Bench-201 as traditionally. To solve this problem, we created a new dataset for the neural evaluator, called "Shallow DARTS". This dataset consists of 100 shallow neural architectures randomly selected from the DARTS search space (i.e., tiny-DARTS space). In order to save training costs, CIFAR-10 was selected as the training dataset. These neural architectures have only 8 layers, which are shallower than the neural architectures reported in Table 2. In addition, we only trained these shallow neural architectures for 50 rounds to further save costs.

[0130] CDP is trained on NAS-Bench-101 and NAS-Bench-201 and used to evaluate the performance of 100 neural architectures in the "Shallow DARTS" dataset. The absolute error between the evaluation value and the true value is not the most important metric, we are more concerned with the relative ranking between neural architectures. Therefore, we use Kendall's Tau (KTau) for our ablation experiments, and we choose it because it is widely used to measure the performance quality of neural evaluators. Furthermore, the value of KTau varies in the range of [-1,1]. The larger the value, the more similar the evaluation ranking is to the actual ranking.

[0131] like Figure 3 As shown in Figure 2, we first performed an ablation experiment on the number of categories C in LMMD. As mentioned earlier, when C is equal to 1, LMMD is mathematically equivalent to MMD. Due to the universal applicability of MMD, the evaluator is improved compared to the baseline method. We then increase C to show the effect of subspace adaptation. When C is set to 2, LMMD performs better than MMD. However, when C is increased to 3, the evaluator does not improve but deteriorates. Further increasing C to 4 leads to a continuous deterioration in performance. This phenomenon may be caused by incorrect pseudo-labels.

[0132] Table 3: Impact of different incremental strategies

[0133]

[0134]

[0135] Table 3 shows the results with and without subspace partitioning. In particular, we specify that the number of subspaces is always 3. In addition, the table also tests the subspace partitioning with a linear increasing rate to show the effectiveness of the cosine function. In general, the performance of the neural evaluator using subspace partitioning is much better. In addition, the cosine function works better than linearly increasing the number of subspaces. Therefore, the ablation experiment proves the effectiveness of the subspace partitioning proposed by this method.

[0136] Table 4: Ablation experiments on the teaching assistant architecture space

[0137] Whether to use the teaching assistant framework space Teaching assistant architecture space size KTau no --- 0.5176 yes Small 0.5051 yes Moderate 0.5306

[0138] Finally, the results of the ablation study of the teaching assistant architecture space are shown in Table 4. The KTau values ​​in the first row are obtained with the help of subspace partitioning (without using the teaching assistant architecture space), which can also be seen from Table 3. On this basis, we add two architecture spaces respectively. The first one is similar to DARTS, and the number of internal nodes is set to 2, which makes its size smaller than the source architecture space and the target architecture space. The second one is "Tiny DARTS", which is the teaching assistant architecture space we proposed, and its size is between the source architecture space and the target architecture space (moderate size). It can be seen from Table 4 that the small teaching assistant architecture space fails to improve the evaluation performance. However, with the help of Tiny DARTS (medium-sized space), the evaluation performance can be further improved based on subspace partitioning.

[0139] In summary, ablation experiments prove that the present invention with subspace partitioning and assistant teaching architecture space (ie, Tiny DARTS) can make the most accurate evaluation.

Claims

1. A cross-domain neural evaluation method, characterized in that: The following steps are involved: S1. Determine the target domain, construct the source architecture space, the assistant architecture space and the target architecture space respectively, record the number of generations as 1, and initialize the feature extraction network and the regression hypothesis model; The source architecture space, the assistant architecture space, and the target architecture space all contain several neural architectures; the neural architectures in the source architecture space are neural architectures with known performance values ​​used in the target domain; the neural architectures in the assistant architecture space and the target architecture space are neural architectures used in other domains; S2, uniformly encode the neural architectures in the source architecture space, the assistant architecture space, and the target architecture space to obtain the encoded source architecture, the encoded assistant architecture, and the encoded target architecture respectively; S3, determine whether the current generation number is less than or equal to the first set value, if so, proceed to step S4; otherwise, proceed to step S11; S4, respectively obtaining the features of the encoded source architecture and the features of the encoded teaching assistant architecture through the current feature extraction network; that is, obtaining the source architecture features and the teaching assistant architecture features respectively; S5. Based on the source architecture features and the teaching assistant architecture features, the performance values ​​of each architecture in the source architecture space and the teaching assistant architecture space are evaluated respectively through the current regression hypothesis model to obtain the performance evaluation values ​​of each architecture in the source architecture space and the teaching assistant architecture space; S6. Update the number of categories of neural architectures in the source architecture space and the assistant architecture space according to the current number of generations; divide the source architecture space and the assistant architecture space into several subspaces according to the number of categories and the performance evaluation value; the number of categories in the subspace where the neural architecture is located is the category value of the neural architecture; S7, obtaining the weight of the corresponding architecture feature based on the category value of the neural architecture in the subspace; S8, performing LMMD calculation on the source architecture features, the teaching assistant architecture features, and the weights of the architecture features to obtain an LMMD calculation value; S9, constructing an objective function according to the performance value of the architecture feature and the corresponding LMMD calculation value, and updating the parameters in the feature extraction network and the regression hypothesis model with the purpose of minimizing the objective function; S10, add 1 to the current generation number, and return to step S3; S11, replacing the encoded teaching assistant architecture with the encoded target architecture, replacing the teaching assistant architecture space with the target architecture space, and updating the parameters in the current feature extraction network and the regression hypothesis model using the same method as steps S3 to S10 until the current number of generations reaches a second set value, and using the current feature extraction network and the regression hypothesis model as the trained cross-domain neural evaluator; S12, using the neural architecture in the target architecture space as the input of the trained cross-domain neural evaluator, evaluating the performance values ​​of all neural architectures in the target architecture space in the target domain, and completing the cross-domain neural evaluation; The target areas in step S1 include: image classification, image recognition, text classification and speech recognition.

2. The cross-domain neural evaluation method according to claim 1, characterized in that: The specific methods for constructing the source architecture space, the assistant architecture space and the target architecture space in step S1 are as follows: The purpose of designing a neural architecture is obtained according to the target domain, and a neural architecture of the same type with known performance values ​​used to achieve the purpose is selected to obtain a source architecture space; Take DARTS space as the target architecture space; The number of intermediate nodes of the architecture in the DARTS space is adjusted from 4 to 3 to obtain the teaching assistant architecture space.

3. The cross-domain neural evaluation method according to claim 1, characterized in that: The specific method of performing unified encoding in step S2 includes the following sub-steps: S2-1. Unify the number of input operations in different neural architectures: adjust the neural architecture with 2 or more input operations into multiple single-input neural architectures by removing input operations; S2-2, obtaining the topological structure of the single-input neural architecture obtained by adjustment; S2-3, obtain the corresponding template adjacency matrix through the topological structure of the single input neural architecture; S2-4, pruning the template adjacency matrix through skip connection and zeroing operation to obtain the preprocessed adjacency matrix; S2-5, filling the penultimate row and penultimate column of the preprocessed adjacency matrix with zeros so that all adjacency matrices after zero filling have the same size, and obtaining an adjacency matrix corresponding to each neural architecture; S2-6, use one-hot encoding to map the input, output, 1×1 convolution, 3×3 convolution, maximum pooling, average pooling and zeroing operations; All zero encoding is used for the remaining operation types; Get the operation encoding matrix corresponding to each neural architecture; Among them, the order in the operation encoding matrix corresponding to the same neural architecture corresponds to the order of each row and column of the adjacency matrix corresponding to the neural architecture; the operation encoding matrix and the adjacency matrix constitute the encoded architecture space.

4. The cross-domain neural evaluation method according to claim 1, characterized in that: The specific method of obtaining the features of the encoded architecture through the feature extraction network in step S4 is: A graph convolutional neural network is established, and a fully connected layer is connected to the output of the graph convolutional neural network; the encoded architecture is used as the input of the graph convolutional neural network, and the output of the fully connected layer is used as the feature of the encoded architecture.

5. The cross-domain neural evaluation method according to claim 1, characterized in that: The specific method of step S6 includes the following sub-steps: S6-1, according to the formula: Update the number of categories of neural architectures in the architecture space ;in K The maximum number of categories is set; is the cosine function; π is 180°; E is the maximum value of the set generation, which is also the second set value; Indicates a round down operation; e is the current generation number; S6-2, sorting the neural architectures in the architecture space from small to large according to their performance values, to obtain a sorted architecture space; S6-3. Divide the sorted architecture space into subspace, will be located in the c The number of categories of neural architectures in the subspace is set to c ;in .

6. The cross-domain neural evaluation method according to claim 1, characterized in that: The specific method of step S7 includes the following sub-steps: S7-1. Initialize a vector with the length of the current number of categories for the label of each neural architecture in the subspace; S7-2, the number of categories is c The vector corresponding to the neural architecture of c Set the first element in the vector to 1 and the remaining elements in the vector to 0 to obtain the value corresponding to the first element in the source architecture space. i The label vector corresponding to the neural architecture , get the first j The label vector corresponding to the neural architecture ; S7-3, according to the formula: Get the source architecture space i The weights of the features corresponding to the neural architecture , and the first j The weights of the features corresponding to the neural architecture ;in Indicates the source schema space i The label vector corresponding to the neural architecture c The value of the element; represents the total number of neural architectures in the source architecture space; Represents the first j The label vector corresponding to the neural architecture c The value of the element; is the total number of neural architectures in the TA architecture space.

7. The cross-domain neural evaluation method according to claim 6, characterized in that: The specific method of step S8 is: According to the formula: Get LMMD calculation value ;in C Indicates the upper limit of the number of subspaces; Indicates the source schema space j The weights of the features corresponding to the neural architecture; Represents the first i The weights of the features corresponding to the neural architecture; is the kernel function; The source architecture feature i element, that is, the first element in the source schema space i The feature corresponding to the neural architecture, that is, the feature i The activation output of the feature extraction network when the neural architecture is input; The source architecture feature j element, that is, the first element in the source schema space j The feature corresponding to the neural architecture, that is, the feature j The activation output of the feature extraction network when the neural architecture is input; The first feature of the teaching assistant architecture j element, i.e., the first element in the teaching assistant architecture space. j The feature corresponding to the neural architecture, that is, the first j The activation output of the feature extraction network when the neural architecture is input; The characteristics of the teaching assistant structure i element, i.e., the first element in the teaching assistant architecture space. i The feature corresponding to the neural architecture, that is, the first i The activation output of the feature extraction network when the neural architecture is input.

8. The cross-domain neural evaluation method according to claim 1, characterized in that: The expression of the objective function in step S9 is: in represents the total number of neural architectures in the source architecture space; is the loss function; W Represents the parameters in the feature extraction network and regression hypothesis model; Indicates the source schema space n A neural architecture; Indicates that the current parameter is W , input is The performance evaluation value when is a constant greater than 0; Calculate the value for LMMD; The source architecture space n The actual performance of the neural architecture.

9. An electronic device, characterized in that: The device comprises: a memory storing executable instructions; and A processor is configured to execute the executable instructions in the memory to implement the method according to any one of claims 1 to 8.

10. A readable storage medium having executable instructions stored thereon, characterized in that: When the executable instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images