Auxiliary learning architecture searching method and device

Abstract

The invention provides an auxiliary learning architecture searching method and device, and the method comprises the steps: building and initializing a connection matrix according to the layer number of a single-task model encoder and the number of alternative fusion operations, and enabling the single-task model to be a model which is predetermined for executing a main task and a plurality of auxiliary tasks; performing normalization processing on a connection weight in the connection matrix by using Gumbel Softmax, and performing weighted fusion on features between the main task and the plurality of auxiliary tasks by using the processed connection weight; and performing regularization constraint on the connection weight from the auxiliary task to the main task, and training until the connection weight from the auxiliary task to the main task converges to be close to zero, thereby obtaining a target network architecture. According to the method, the fusion operation search space with multiple selectable fusion operations is created, so that the model can dynamically adjust the fusion strategy according to different sample characteristics.

Description

Technical Field

[0001] The present invention relates to the field of neural architecture search technology, and in particular to a method and device for auxiliary learning architecture search. Background Art

[0002] In the field of deep learning, with the explosive growth of data volume and the requirements for model generalization and high performance, multi-task learning has emerged. The core idea of ​​this learning method is to let the model share representations and knowledge during the learning process. By sharing parameters or feature representations, the model can transfer knowledge between multiple tasks, thereby improving overall performance. However, multi-task learning often increases the demand for computing resources, and the convergence speed of different tasks may be different. Tasks that converge slowly are likely to have underfitting and other phenomena. The emergence of auxiliary learning cleverly resolves these drawbacks.

[0003] Auxiliary learning is a special case of multi-task learning, where the main focus is on the performance of one or a few tasks, while the other tasks exist as auxiliary tasks. In practical applications, auxiliary learning can be performed by setting auxiliary tasks with various attributes, which help us understand the correlation between tasks and improve the performance of the main task.

[0004] However, an efficient multi-task network structure must take into account both the feature sharing part and the task-specific part, and be able to reasonably measure and select auxiliary task features that are beneficial to the main task. Therefore, how to reasonably select the neural network architecture for auxiliary learning has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The present invention provides a method and device for searching an auxiliary learning architecture, which are used to solve the defect of the prior art in the lack of a search method for an auxiliary learning neural network architecture, and realize an efficient and flexible search method for an auxiliary learning neural network architecture.

[0006] The present invention provides an auxiliary learning architecture search method, comprising: Constructing and initializing a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and a plurality of auxiliary tasks; Using Gumbel Softmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion on the features of the main task and several auxiliary tasks; Regularization constraints are applied to the connection weights from the auxiliary task to the main task, and when the connection weights from the auxiliary task to the main task are trained to converge to near zero, a target network architecture is obtained.

[0007] According to an auxiliary learning architecture search method provided by the present invention, the connection matrix is ​​a three-dimensional connection matrix of N×N×S, wherein N is the number of encoder layers of the single-task model, and S is the number of alternative fusion operations; the first dimension of the connection matrix represents the encoder layer number where the starting position of the feature fusion is located, the second dimension represents the encoder layer number where the target position of the feature fusion is located, and the third dimension represents the fusion operation used by the current connection point.

[0008] According to an auxiliary learning architecture search method provided by the present invention, the alternative fusion operations include neural discriminant dimensionality reduction (NDDR) operations with a convolution kernel size of 1×1, a convolution kernel size of 3×3, and a convolution kernel size of 5×5.

[0009] According to an auxiliary learning architecture search method provided by the present invention, the step of using Gumbel Softmax to normalize the connection weights in the connection matrix specifically includes: Adding Gumbel noise to the connection weights in the connection matrix and then scaling them; Apply a softmax function to the connection weights in the scaled connection matrix.

[0010] According to an auxiliary learning architecture search method provided by the present invention, before the step of constructing and initializing a connection matrix according to the number of encoder layers of the single-task model and the number of candidate fusion operations, the method further includes: Decomposing the encoder of the single-task model into separate encoder layers; Extract the input features and output features of a single task at each of the encoder layers.

[0011] According to an auxiliary learning architecture search method provided by the present invention, the loss function for regularizing the connection weights from the auxiliary task to the main task is determined based on the loss of the main task, the loss of the auxiliary task and the regularization term.

[0012] The present invention also provides an auxiliary learning architecture search device, comprising: A construction module, configured to construct and initialize a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and a plurality of auxiliary tasks; A fusion module, used to use Gumbel Softmax to normalize the connection weights in the connection matrix, and use the processed connection weights to perform weighted fusion on the features of the main task and several auxiliary tasks; A determination module is used to perform regularization constraints on the connection weights from the auxiliary task to the main task, and obtain a target network architecture when the connection weights from the auxiliary task to the main task are trained to converge to near zero.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the auxiliary learning architecture search method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described auxiliary learning architecture search methods.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned auxiliary learning architecture search methods.

[0016] The auxiliary learning architecture search method and device provided by the present invention can enable the model to dynamically adjust the fusion strategy according to different sample features by creating a search space with multiple optional fusion operations, and use Gumbel Softmax to achieve distribution modeling of discrete weight parameters. Through continuous optimization during the training process, it is ensured that the network can autonomously select the most appropriate convolution operation in each feature fusion execution process. At the same time, the learnable parameters are divided into model parameters and architecture parameters for simultaneous training, which improves the operation efficiency and flexibility of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flowchart of the auxiliary learning architecture search method provided by the present invention; Figure 2 It is a schematic diagram of the feature fusion process of the auxiliary learning architecture search method provided by the present invention; Figure 3 It is a schematic diagram of processing a connection matrix using Gumbel Softmax in the auxiliary learning architecture search method provided by the present invention; Figure 4 It is a structural schematic diagram of the auxiliary learning architecture search device provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Combine the following Figures 1 to 3 The invention introduces the auxiliary learning architecture search method, such as Figure 1 As shown, including: Step 101, constructing and initializing a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and a plurality of auxiliary tasks; It is understandable that as a kind of multi-task learning, an efficient auxiliary learning neural network architecture must take into account both the feature sharing part and the task-specific part, and be able to reasonably measure and select auxiliary task features that are beneficial to the main task. Therefore, it becomes very important to design a high-performance neural network architecture.

[0021] Traditional neural network architecture design relies on expert experience and a large number of experiments, which is time-consuming and lacks systematicity. In recent years, with the development of neural architecture search (NAS) technology, it has become possible to automatically design network architecture.

[0022] NAS uses algorithms to automatically explore different network structures to find the most effective model for the target task. NAS defines a search space containing a variety of possible network architectures, and then uses intelligent search strategies to search in the space and evaluate the performance of different architectures for iterative optimization.

[0023] Therefore, to search for auxiliary learning architectures, we first need to determine the search space of NAS based on the main task and auxiliary tasks.

[0024] Optionally, one main task may correspond to one auxiliary task or multiple auxiliary tasks.

[0025] Optionally, a single-task model for performing the task is determined according to the categories of the main task and the auxiliary task, and a search space is constructed based on the selected single-task model.

[0026] In this implementation, the semantic segmentation of an image is used as the main task and the plane normal prediction is used as the auxiliary task. It can be understood that since the main task and the auxiliary task selected in this implementation are both image processing tasks, ViTBase is selected as the backbone network of a single image task (main task or auxiliary task), that is, ViTBase is determined as a single-task model.

[0027] ViTBase uses the Transformer architecture. The input image is first divided into multiple small blocks (patches), and each patch is regarded as a "token". These patches are embedded in a high-dimensional space and then input into the Transformer model. The ViTBase model uses the self-attention mechanism to capture the relationship between different areas in the image. The model is usually pre-trained on a large-scale image dataset and then fine-tuned on a specific task. Pre-training can help the model learn general visual features, while fine-tuning enables the model to focus on specific tasks.

[0028] It should be noted that the encoder of the selected single-task model needs to be able to be layered, that is, its encoder layer can be decomposed into step-by-step calculations as the basis for architecture search.

[0029] Furthermore, since the auxiliary learning architecture is mainly used to measure and select auxiliary task features that are beneficial to the main task, after determining the single-task model, the model framework is first organized.

[0030] Taking the ViTBase model as an example, we first package image segmentation (patches), linear embedding, position encoding, and CLS insertion into base operations, that is, encapsulate the modules before the encoder, then decompose the encoder layer into step-by-step calculations, and finally connect the decoders to different task heads respectively.

[0031] In addition, in order to integrate the image's shallow detail information and deep semantic information, a neural network structure based on residual convolution units and fusion modules is used in the single-task model to process image data. The former is used to process the input feature map, reassemble the feature maps from different stages into a complete feature map, and keep the spatial resolution of the feature map unchanged; the latter is used to fuse the feature maps from different stages and perform upsampling. A Reassemble and a Fusion module are created for each stage through a cycle and added to their respective lists so that these modules can be called correctly during the forward propagation of the model.

[0032] Furthermore, neural network head modules are designed to process normal maps and segmented images respectively. These modules include a series of convolutional layers, interpolation layers and activation functions, which serve as decoders to decode the feature maps generated by ViT into the final output.

[0033] Based on the above steps, build a search space for feature fusion operations.

[0034] First, multiple candidate fusion operations are determined, and the candidate fusion operations represent the method of feature fusion.

[0035] Optionally, the alternative fusion operations may include feature fusion methods such as concatenation, addition, and pooling.

[0036] In the auxiliary learning architecture search method of the present invention, the alternative fusion operations include neural discriminant dimensionality reduction NDDR operations with a convolution kernel size of 1×1, a convolution kernel size of 3×3, and a convolution kernel size of 5×5.

[0037] Specifically, NDDR (Neural Discriminative Dimensionality Reduction) is a convolutional neural network structure for multi-task learning. NDDR first concatenates the features of the same spatial resolution of different tasks according to the channel dimension, and then achieves dimensionality reduction through convolution operations, batch normalization and weight decay, which helps to reduce the dimension of features while retaining key information, so that the fused features meet the input requirements of subsequent network layers. The effect of neural architecture search is achieved by setting different convolution kernel sizes.

[0038] In other feasible implementations, the alternative fusion operation may also be NDDR of 7×7 conv, or an operation of disconnecting.

[0039] After determining the alternative fusion operations, the connection matrix is ​​constructed and initialized according to the number of encoder layers of the single-task model and the number of alternative fusion operations. It can be understood that the connection matrix represents the search range of the auxiliary learning architecture. That is to say, this search is to determine the best feature fusion method for each encoder layer of the single-task model among multiple alternative fusion operations, so as to obtain the final auxiliary learning neural network architecture.

[0040] Before searching, the connection matrix is ​​initialized, that is, the connection weights in the connection matrix are initialized to the same initial value, which is 1 in this embodiment.

[0041] Determine the maximum number of fusion layers and the location of the connection points of the single-task model to ensure that the input feature position of the i-th layer only receives features from the i-3, i-2, and i-1 layers of another task.

[0042] Step 102, using Gumbel Softmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion on the features of the main task and several auxiliary tasks; The study found that during the NAS search process, if the architecture parameters are not constrained or are only limited to a range between 0 and 1, the final architecture parameters will be concentrated on a certain operation, which is not conducive to the model exploring more diverse and rich combination operations.

[0043] It can be understood that the architecture parameters represent the parameters of the neural network search architecture, which in this embodiment are the relevant parameters of the connection matrix.

[0044] Therefore, in order to constrain the multivariate parameters and optimize the model convergence results, Gumbel Softmax is used to normalize the connection weights in the connection matrix, and then the processed connection weights are used to perform weighted fusion on the single task features.

[0045] It should be noted that when constraining multivariate parameters, the softmax function is usually used to understand the architecture parameters as the probability distribution corresponding to each operation. However, since the connection matrix constructed in this implementation is initialized to be equal, using softmax to normalize it will affect the parameter convergence speed. To this end, we further explored the effects of conditions such as randomly initializing architecture parameters, limiting the upper and lower limits of parameters (torch.clamp function) or not constraining them on the model convergence results. Finally, we chose to introduce Gumbel Softmax to first introduce noise to the parameters and then normalize them. By introducing uncertainty, the amplitude of the connection weight fluctuation is increased, so that the model can explore a wider space, and the final model convergence effect is also better.

[0046] Step 103, regularizing the connection weights from the auxiliary task to the main task, and obtaining the target network architecture when the connection weights from the auxiliary task to the main task converge to zero during training.

[0047] Finally, set the optimizer type used by the single-task model, define the loss function based on the neural network head module of the main task and each auxiliary task, and use the back-propagation gradient to update the model parameters and architecture parameters.

[0048] Among them, the model parameters are the parameters of the selected single-task model; the architecture parameters represent the parameters of the neural network search architecture.

[0049] like Figure 2As shown, the training process of this embodiment can be understood as initializing bidirectional connections and guiding NAS optimization in the search space, and using regularization constraints to make the optimization process converge to an asymmetric network architecture with only a unidirectional connection from the main task to the auxiliary task. This allows the connection from the auxiliary task to the main task to be safely removed during the main task reasoning, thereby achieving multi-task level performance and single-task reasoning cost.

[0050] That is to say, when the connection weights from the auxiliary task to the main task converge to near zero during training, the model is considered to have been optimized into an asymmetric network architecture with only a one-way connection from the main task to the auxiliary task, and it is determined as the auxiliary task architecture searched, that is, the target network architecture, and the alternative fusion operations corresponding to each encoder layer at this time are determined.

[0051] The present invention creates a search space with multiple fusion operations to choose from, which enables the model to dynamically adjust the fusion strategy according to different sample features, and uses Gumbel Softmax to achieve distribution modeling of discrete weight parameters. Through continuous optimization during the training process, it ensures that the network can autonomously select the most appropriate convolution operation in each feature fusion execution process. At the same time, the learnable parameters are divided into model parameters and architecture parameters for simultaneous training, which improves the operation efficiency and flexibility of the algorithm.

[0052] In the auxiliary learning architecture search method of the present invention, the connection matrix is ​​a three-dimensional connection matrix of N×N×S, wherein N is the number of encoder layers of the single-task model, and S is the number of alternative fusion operations; the first dimension of the connection matrix represents the encoder layer number where the starting position of the feature fusion is located, the second dimension represents the encoder layer number where the target position of the feature fusion is located, and the third dimension represents the fusion operation used by the current connection point.

[0053] Taking the ViTBase model as a single-task model as an example, the three-dimensional connection matrix in this implementation is explained. According to the number of encoder layers N of ViTBase and the number of elements S in the search space, a three-dimensional connection matrix of size N×N×S is constructed and all its element values ​​are initialized to the same preset value, which is set to 1 in this implementation.

[0054] Among them, the elements of the search space are the candidate fusion operations.

[0055] It can be understood that the number of layers of the encoder is different for different determined single-task models; the number of alternative fusion operations is different for different determined alternative fusion operations, and the constructed three-dimensional connection matrix is ​​also adjusted accordingly.

[0056] In the auxiliary learning architecture search method of the present invention, the step of using Gumbel Softmax to normalize the connection weights in the connection matrix specifically includes: Adding Gumbel noise to the connection weights in the connection matrix and then scaling them; Apply a softmax function to the connection weights in the scaled connection matrix.

[0057] In order to achieve the purpose of NAS, the most appropriate fusion strategy is selected for each connection point, and at the same time, the gradient of the weight parameter is ensured to exist. In this embodiment of the present invention, Gumbel Softmax is used to process the weight parameter.

[0058] Specifically, Figure 3 As shown, first add Gumbel noise to the connection weight parameters of the connection matrix to make it random. Gumbel noise is sampled from the Gumbel distribution, which is an extreme value distribution used to simulate the distribution formed by extreme values ​​sampled from other distributions. The formula for Gumbel noise is: ; In the formula, From a uniform distribution A random variable sampled from .

[0059] The weight parameters with added noise are then scaled and the Softmax function is applied so that the output is close to the one-hot vector but maintains continuity so that gradient calculation can be performed. The Softmax formula is: ; In the formula, is a temperature parameter used to control the smoothness of the output. When it approaches 0, the output is close to a one-hot vector. As it approaches infinity, the output approaches a uniform distribution.

[0060] Gumbel Softmax uses the reparameterization technique to transfer the non-differentiable sampling process to Gumbel noise, which itself does not depend on the original probability distribution, so the entire sampling process becomes differentiable. In this way, the model parameters can be updated through back propagation, achieving the differentiability of discrete variables.

[0061] On this basis, the weight parameters after Gumbel Softmax processing have the characteristics of one-hot distribution, thus realizing the fusion operation selection of NAS. These parameters are multiplied by the features of the corresponding connection points to achieve feature weighting. The feature fusion process formula at the i-th layer of the model is as follows: ; In the formula, P i and A i Respectively represent i The main task features and auxiliary task features of the layer output, is the architecture parameter of the network. The superscript P indicates that this is the connection from the auxiliary task to the main task, the subscripts i and j indicate that this parameter is the connection weight from the i-th layer of the auxiliary task to the j-th layer of the main task, and 0, 1, and 2 correspond to three determined alternative fusion operations.

[0062] After performing weighted fusion, normalization, and activation operations, the softmax function is applied to generate a smooth approximate discrete distribution, and the obtained features of this layer are used as the input of the i+1th layer encoder to continue forward propagation.

[0063] In the auxiliary learning architecture search method of the present invention, before the step of constructing and initializing the connection matrix according to the number of encoder layers of the single-task model and the number of candidate fusion operations, the method further includes: Decomposing the encoder of the single-task model into separate encoder layers; Extract the input features and output features of a single task at each of the encoder layers.

[0064] In order to construct the NAS search space, it is necessary to split the encoder of the single-task model in advance and determine the input features and output features of each layer of the encoder obtained by the split so as to calculate them when searching the architecture.

[0065] Taking the ViTBase model as an example, the 12 stages of its encoder layer are broken down into step-by-step calculations, and loops are used to extract the input and output features of a single task in each encoder layer (stage), realizing layer-by-layer operations on the data flow and preparing conditions for feature fusion between tasks in subsequent steps.

[0066] In the auxiliary learning architecture search method of the present invention, the loss function for regularizing the connection weights from the auxiliary task to the main task is determined based on the loss of the main task, the loss of the auxiliary task and the regularization term.

[0067] In this implementation, the loss function during architecture search is determined based on the loss of the main task, the loss of the auxiliary task, and the regularization term, as shown below: ; In the formula, and They correspond to the loss function of the main task head and the loss function of the auxiliary task head respectively. The regularization constraint added to the connection weights of the auxiliary task to the main task is .

[0068] The present invention designs a flexible asymmetric network structure, which can use different networks in the training stage and the inference stage, and avoids negative transfer between different tasks by sharing soft parameters and learning the independent features of each task. At the same time, regularization is introduced to guide NAS optimization to converge to a connection architecture with only one-sided main task to auxiliary task, thereby realizing the inference cost of a single task in the inference stage, and further improving the performance of auxiliary learning and architecture search technology.

[0069] Furthermore, the comparison results of the auxiliary learning architecture search method of the present invention and the reasoning results of a single task are shown in Table 1 below.

[0070] Table 1

[0071] The task corresponding to P on the left side of the table is the main task; the task corresponding to A is the auxiliary task. The bold number on the right is the optimal indicator. It can be seen that when the selected single-task backbone network is ViTBase, the data set used is NYU-v2, and the two tasks are set to image segmentation and plane normal prediction respectively, whether it is the main task or the auxiliary task, the running results constructed using the algorithm in this implementation have better performance.

[0072] The auxiliary learning architecture search device provided by the present invention is described below. The auxiliary learning architecture search device described below and the auxiliary learning architecture search method described above can be referenced to each other.

[0073] like Figure 4 As shown, the auxiliary learning architecture search device provided by the present invention includes a construction module 401, a fusion module 402 and a determination module 403: A construction module 401 is used to construct and initialize a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and a plurality of auxiliary tasks; It is understandable that as a kind of multi-task learning, an efficient auxiliary learning neural network architecture must take into account both the feature sharing part and the task-specific part, and be able to reasonably measure and select auxiliary task features that are beneficial to the main task. Therefore, it becomes very important to design a high-performance neural network architecture.

[0074] Traditional neural network architecture design relies on expert experience and a large number of experiments, which is time-consuming and lacks systematicity. In recent years, with the development of neural architecture search (NAS) technology, it has become possible to automatically design network architecture.

[0075] NAS uses algorithms to automatically explore different network structures to find the most effective model for the target task. NAS defines a search space containing a variety of possible network architectures, and then uses intelligent search strategies to search in the space and evaluate the performance of different architectures for iterative optimization.

[0076] Therefore, to search for auxiliary learning architectures, we first need to determine the search space of NAS based on the main task and auxiliary tasks.

[0077] Optionally, one main task may correspond to one auxiliary task or multiple auxiliary tasks.

[0078] Optionally, a single-task model for performing the task is determined according to the categories of the main task and the auxiliary task, and a search space is constructed based on the selected single-task model.

[0079] In this implementation, the semantic segmentation of an image is used as the main task and the plane normal prediction is used as the auxiliary task. It can be understood that since the main task and the auxiliary task selected in this implementation are both image processing tasks, ViTBase is selected as the backbone network of a single image task (main task or auxiliary task), that is, ViTBase is determined as a single-task model.

[0080] ViTBase uses the Transformer architecture. The input image is first divided into multiple small blocks (patches), and each patch is regarded as a "token". These patches are embedded in a high-dimensional space and then input into the Transformer model. The ViTBase model uses the self-attention mechanism to capture the relationship between different areas in the image. The model is usually pre-trained on a large-scale image dataset and then fine-tuned on a specific task. Pre-training can help the model learn general visual features, while fine-tuning enables the model to focus on specific tasks.

[0081] It should be noted that the encoder of the selected single-task model needs to be able to be layered, that is, its encoder layer can be decomposed into step-by-step calculations as the basis for architecture search.

[0082] Furthermore, since the auxiliary learning architecture is mainly used to measure and select auxiliary task features that are beneficial to the main task, after determining the single-task model, the model framework is first organized.

[0083] Taking the ViTBase model as an example, we first package image segmentation (patches), linear embedding, position encoding, and CLS insertion into base operations, that is, encapsulate the modules before the encoder, then decompose the encoder layer into step-by-step calculations, and finally connect the decoders to different task heads respectively.

[0084] In addition, in order to integrate the image's shallow detail information and deep semantic information, a neural network structure based on residual convolution units and fusion modules is used in the single-task model to process image data. The former is used to process the input feature map, reassemble the feature maps from different stages into a complete feature map, and keep the spatial resolution of the feature map unchanged; the latter is used to fuse the feature maps from different stages and perform upsampling. A Reassemble and a Fusion module are created for each stage through a cycle and added to their respective lists so that these modules can be called correctly during the forward propagation of the model.

[0085] Furthermore, neural network head modules are designed to process normal maps and segmented images respectively. These modules include a series of convolutional layers, interpolation layers and activation functions, which serve as decoders to decode the feature maps generated by ViT into the final output.

[0086] Based on the above steps, build a search space for feature fusion operations.

[0087] First, multiple candidate fusion operations are determined, and the candidate fusion operations represent the method of feature fusion.

[0088] Optionally, the alternative fusion operations may include feature fusion methods such as concatenation, addition, and pooling.

[0089] After determining the alternative fusion operations, the connection matrix is ​​constructed and initialized according to the number of encoder layers of the single-task model and the number of alternative fusion operations. It can be understood that the connection matrix represents the search range of the auxiliary learning architecture. That is to say, this search is to determine the best feature fusion method for each encoder layer of the single-task model among multiple alternative fusion operations, so as to obtain the final auxiliary learning neural network architecture.

[0090] A fusion module 402 is used to use Gumbel Softmax to normalize the connection weights in the connection matrix, and use the processed connection weights to perform weighted fusion on the features of the main task and the plurality of auxiliary tasks; The study found that during the NAS search process, if the architecture parameters are not constrained or are only limited to a range between 0 and 1, the final architecture parameters will be concentrated on a certain operation, which is not conducive to the model exploring more diverse and rich combination operations.

[0091] It can be understood that the architecture parameters represent the parameters of the neural network search architecture, which in this embodiment are the relevant parameters of the connection matrix.

[0092] Therefore, in order to constrain the multivariate parameters and optimize the model convergence results, Gumbel Softmax is used to normalize the connection weights in the connection matrix, and then the processed connection weights are used to perform weighted fusion on the single task features.

[0093] It should be noted that when constraining multivariate parameters, the softmax function is usually used to understand the architecture parameters as the probability distribution corresponding to each operation. However, since the connection matrix constructed in this implementation is initialized to be equal, using softmax to normalize it will affect the parameter convergence speed. To this end, we further explored the effects of conditions such as randomly initializing architecture parameters, limiting the upper and lower limits of parameters (torch.clamp function) or not constraining them on the model convergence results. Finally, we chose to introduce Gumbel Softmax to first introduce noise to the parameters and then normalize them. By introducing uncertainty, the amplitude of the connection weight fluctuation is increased, so that the model can explore a wider space, and the final model convergence effect is also better.

[0094] The determination module 403 is used to perform regularization constraints on the connection weights from the auxiliary task to the main task, and obtain the target network architecture when the connection weights from the auxiliary task to the main task are trained to converge to zero.

[0095] Finally, set the optimizer type used by the single-task model, define the loss function based on the neural network head module of the main task and each auxiliary task, and use the back-propagation gradient to update the model parameters and architecture parameters.

[0096] Among them, the model parameters are the parameters of the selected single-task model; the architecture parameters represent the parameters of the neural network search architecture.

[0097] like Figure 2 As shown, the training process of this embodiment can be understood as initializing bidirectional connections and guiding NAS optimization in the search space, and using regularization constraints to make the optimization process converge to an asymmetric network architecture with only a unidirectional connection from the main task to the auxiliary task. This allows the connection from the auxiliary task to the main task to be safely removed during the main task reasoning, thereby achieving multi-task level performance and single-task reasoning cost.

[0098] That is to say, when the connection weights from the auxiliary task to the main task converge to near zero during training, the model is considered to have been optimized into an asymmetric network architecture with only a one-way connection from the main task to the auxiliary task, and it is determined as the auxiliary task architecture searched, that is, the target network architecture, and the alternative fusion operations corresponding to each encoder layer at this time are determined.

[0099] The present invention creates a search space with multiple fusion operations to choose from, which enables the model to dynamically adjust the fusion strategy according to different sample features, and uses Gumbel Softmax to achieve distribution modeling of discrete weight parameters. Through continuous optimization during the training process, it ensures that the network can autonomously select the most appropriate convolution operation in each feature fusion execution process. At the same time, the learnable parameters are divided into model parameters and architecture parameters for simultaneous training, which improves the operation efficiency and flexibility of the algorithm.

[0100] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the auxiliary learning architecture search method, which includes: constructing and initializing a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for executing a main task and several auxiliary tasks; using Gumbel Softmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion on the features between the main task and several auxiliary tasks; regularizing the connection weights from the auxiliary task to the main task, and obtaining the target network architecture when the connection weights from the auxiliary task to the main task converge to zero during training.

[0101] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0102] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the auxiliary learning architecture search method provided by the above methods, the method including: constructing and initializing a connection matrix according to the number of encoder layers of a single-task model and the number of alternative fusion operations, wherein the single-task model is a model predetermined for executing a main task and several auxiliary tasks; using Gumbel Softmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks; regularizing the connection weights from the auxiliary task to the main task, and obtaining the target network architecture when the connection weights from the auxiliary task to the main task converge to near zero during training.

[0103] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the auxiliary learning architecture search method provided by the above-mentioned methods, the method comprising: constructing and initializing a connection matrix according to the number of encoder layers of a single-task model and the number of alternative fusion operations, wherein the single-task model is a model predetermined for executing a main task and several auxiliary tasks; using GumbelSoftmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks; regularizing the connection weights from the auxiliary task to the main task, and obtaining the target network architecture when the connection weights from the auxiliary task to the main task converge to near zero during training.

[0104] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0105] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0106] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for searching an auxiliary learning architecture, characterized in that: include: Constructing and initializing a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and a plurality of auxiliary tasks; Using Gumbel Softmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion on the features of the main task and several auxiliary tasks; Regularization constraints are applied to the connection weights from the auxiliary task to the main task, and when the connection weights from the auxiliary task to the main task are trained to converge to near zero, a target network architecture is obtained.

2. The auxiliary learning architecture search method according to claim 1, characterized in that: The connection matrix is ​​a three-dimensional connection matrix of N×N×S, wherein N is the number of encoder layers of the single-task model, and S is the number of the alternative fusion operations; the first dimension of the connection matrix represents the encoder layer number where the starting position of the feature fusion is located, the second dimension represents the encoder layer number where the target position of the feature fusion is located, and the third dimension represents the fusion operation used by the current connection point.

3. The auxiliary learning architecture search method according to claim 1, characterized in that: The alternative fusion operations include neural discriminant dimensionality reduction (NDDR) operations with convolution kernel size of 1×1, convolution kernel size of 3×3, and convolution kernel size of 5×5.

4. The auxiliary learning architecture search method according to any one of claims 1 to 3, characterized in that: The step of using Gumbel Softmax to normalize the connection weights in the connection matrix specifically includes: Adding Gumbel noise to the connection weights in the connection matrix and then scaling them; Apply a softmax function to the connection weights in the scaled connection matrix.

5. The auxiliary learning architecture search method according to any one of claims 1 to 3, characterized in that: Before the step of constructing and initializing the connection matrix according to the number of encoder layers of the single-task model and the number of candidate fusion operations, the method further includes: Decomposing the encoder of the single-task model into separate encoder layers; Extract the input features and output features of a single task at each of the encoder layers.

6. The auxiliary learning architecture search method according to any one of claims 1 to 3, characterized in that: The loss function for regularizing the connection weights from the auxiliary task to the main task is determined according to the loss of the main task, the loss of the auxiliary task and the regularization term.

7. An auxiliary learning architecture search device, characterized in that: include: A construction module, configured to construct and initialize a connection matrix according to the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and a plurality of auxiliary tasks; A fusion module, used to use Gumbel Softmax to normalize the connection weights in the connection matrix, and use the processed connection weights to perform weighted fusion on the features of the main task and several auxiliary tasks; A determination module is used to perform regularization constraints on the connection weights from the auxiliary task to the main task, and obtain a target network architecture when the connection weights from the auxiliary task to the main task are trained to converge to near zero.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the auxiliary learning architecture search method as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the auxiliary learning architecture search method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the auxiliary learning architecture search method according to any one of claims 1 to 6 is implemented.

Images