Auxiliary learning architecture search method and device

Abstract

The application provides an auxiliary learning architecture search method and device, which comprises the following steps: constructing and initializing a connection matrix according to the number of single-task model encoder layers and the number of alternative fusion operations, wherein the single-task model is a model determined in advance for performing a main task and a plurality of auxiliary tasks; using Gumbel Softmax to normalize the connection weights in the connection matrix, and using the processed connection weights to perform weighted fusion on the features between the main task and the plurality of auxiliary tasks; and performing regularization constraint on the connection weights from the auxiliary task to the main task, and obtaining a target network architecture in the case that the connection weights from the auxiliary task to the main task converge to a value close to zero after training. By creating a search space with a plurality of alternative fusion operations, the model can dynamically adjust the fusion strategy according to different sample features.

Description

Technical Field

[0001] This invention relates to the field of neural architecture search technology, and in particular to a method and apparatus for assisting in the search of learning architectures. Background Technology

[0002] In the field of deep learning, with the explosive growth of data and the demand for model generalization and high performance, multi-task learning has emerged. The core idea of ​​this learning approach is to allow models to share representations and knowledge during the learning process. By sharing parameters or feature representations, models can transfer knowledge between multiple tasks, thereby improving overall performance. However, multi-task learning often increases the demand for computational resources, and different tasks may converge at different speeds. Tasks with slower convergence may experience underfitting. The emergence of assisted learning cleverly resolves these drawbacks.

[0003] Assisted learning is a special case of multi-task learning, where the primary focus is on performance on one or a few tasks, while other tasks serve as auxiliary tasks. In practical applications, assisted learning can be implemented by setting auxiliary tasks with various attributes. These auxiliary tasks help us understand the correlations between tasks and improve the performance of the main task.

[0004] However, an efficient multi-task network structure must simultaneously consider both feature sharing and task-specific aspects, and be able to reasonably measure and select auxiliary task features that are beneficial to the main task. Therefore, how to reasonably select the neural network architecture for auxiliary learning has become an urgent technical problem to be solved. Summary of the Invention

[0005] This invention provides a method and apparatus for searching assisted learning architectures, which addresses the shortcomings of existing methods for searching assisted learning neural network architectures and provides an efficient and flexible method for searching assisted learning neural network architectures.

[0006] This invention provides a method for assisting in learning architecture search, comprising:

[0007] The connection matrix is ​​constructed and initialized based on the number of encoder layers in the single-task model and the number of candidate fusion operations, wherein the single-task model is a model pre-determined for executing the main task and several auxiliary tasks;

[0008] The connection weights in the connection matrix are normalized using Gumbel Softmax, and the processed connection weights are used to perform weighted fusion of features between the main task and several auxiliary tasks.

[0009] The connection weights from the auxiliary task to the main task are subject to regularization constraints. The target network architecture is obtained when the connection weights from the auxiliary task to the main task converge to near zero during training.

[0010] According to the auxiliary learning architecture search method provided by the present invention, the connection matrix is ​​a three-dimensional connection matrix of N×N×S, where N is the number of encoder layers of the single task model and S is the number of candidate fusion operations; the first dimension of the connection matrix represents the number of encoder layers where the starting position of feature fusion is located, the second dimension represents the number of encoder layers where the target position of feature fusion is located, and the third dimension represents the fusion operation used at the current connection point.

[0011] According to the auxiliary learning architecture search method provided by the present invention, the alternative fusion operation includes neural discrimination dimensionality reduction (NDDR) operations with convolution kernel sizes of 1×1, 3×3, and 5×5.

[0012] According to the auxiliary learning architecture search method provided by the present invention, the step of normalizing the connection weights in the connection matrix using Gumbel Softmax specifically includes:

[0013] The connection weights in the connection matrix are scaled after adding Gumbel noise.

[0014] Apply the softmax function to the connection weights in the scaled connection matrix.

[0015] According to the auxiliary learning architecture search method provided by the present invention, before the step of constructing and initializing the connection matrix based on the number of encoder layers of the single-task model and the number of candidate fusion operations, the method further includes:

[0016] The encoder of the single-task model is decomposed into a separate encoder layer;

[0017] Extract the input and output features of a single task at each encoder layer.

[0018] According to the auxiliary learning architecture search method provided by the present invention, the loss function for regularizing the connection weights from the auxiliary task to the main task is determined based on the loss of the main task, the loss of the auxiliary task, and the regularization term.

[0019] The present invention also provides an auxiliary learning architecture search device, comprising:

[0020] A construction module is used to construct and initialize a connection matrix based on the number of encoder layers in a single-task model and the number of alternative fusion operations, wherein the single-task model is a model pre-determined for executing the main task and several auxiliary tasks.

[0021] The fusion module is used to normalize the connection weights in the connection matrix using Gumbel Softmax, and then use the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks.

[0022] The determination module is used to apply regularization constraints to the connection weights from the auxiliary task to the main task. When the connection weights from the auxiliary task to the main task converge to near zero during training, the target network architecture is obtained.

[0023] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the assisted learning architecture search method as described above.

[0024] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the auxiliary learning architecture search method as described above.

[0025] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the assisted learning architecture search method as described above.

[0026] The auxiliary learning architecture search method and apparatus provided by this invention create a search space with multiple selectable fusion operations, enabling the model to dynamically adjust the fusion strategy according to different sample features. It uses Gumbel Softmax to model the distribution of discrete weight parameters, and through continuous optimization during training, ensures that the network can autonomously select the most suitable convolution operation in each feature fusion execution. Simultaneously, the learnable parameters are trained separately as model parameters and architecture parameters, improving the algorithm's efficiency and flexibility. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0028] Figure 1 This is a flowchart illustrating the assisted learning architecture search method provided by the present invention;

[0029] Figure 2 This is a schematic diagram of the feature fusion process of the assisted learning architecture search method provided by the present invention;

[0030] Figure 3 This is a schematic diagram illustrating the processing of the connection matrix using Gumbel Softmax in the assisted learning architecture search method provided by this invention;

[0031] Figure 4 This is a schematic diagram of the structure of the auxiliary learning architecture search device provided by the present invention;

[0032] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0034] The following is combined Figures 1 to 3 This invention introduces the assisted learning architecture search method, such as... Figure 1 As shown, it includes:

[0035] Step 101: Construct and initialize the connection matrix based on the number of encoder layers in the single-task model and the number of candidate fusion operations, wherein the single-task model is a model pre-determined for executing the main task and several auxiliary tasks.

[0036] Understandably, as a form of multi-task learning, an efficient auxiliary learning neural network architecture must simultaneously consider both the feature-sharing part and the task-specific part, and be able to reasonably measure and select auxiliary task features that are beneficial to the main task. Therefore, designing a high-performance neural network architecture becomes very important.

[0037] Traditional neural network architecture design relies on expert experience and extensive experimentation, a process that is both time-consuming and lacks systematic approach. In recent years, however, the development of Neural Architecture Search (NAS) technology has made automated network architecture design possible.

[0038] NAS (Network Attached Search) uses algorithms to automatically explore different network architectures in order to discover the most effective model for the target task. NAS defines a search space containing multiple possible network architectures, and then uses intelligent search strategies to search within the space and evaluate the performance of different architectures for iterative optimization.

[0039] Therefore, the first step in developing a search-assisted learning architecture is to determine the search space of the NAS based on the primary and secondary tasks.

[0040] Optionally, a main task can correspond to one auxiliary task or multiple auxiliary tasks.

[0041] Optionally, a single-task model for the task to be performed can be determined based on the categories of the main task and auxiliary tasks, and a search space can be constructed based on the selected single-task model.

[0042] In this embodiment, semantic segmentation of an image is taken as the primary task and plane normal prediction is taken as the auxiliary task for illustration. It can be understood that since both the primary and auxiliary tasks selected in this embodiment are image processing tasks, ViTBase is selected as the backbone network for a single image task (primary task or auxiliary task), that is, ViTBase is determined as a single-task model.

[0043] ViTBase employs a Transformer architecture. The input image is first segmented into multiple patches, each treated as a "token." These patches are embedded into a high-dimensional space and then fed into the Transformer model. The ViTBase model utilizes a self-attention mechanism to capture the relationships between different regions in the image. This model is typically pre-trained on large-scale image datasets and then fine-tuned for specific tasks. Pre-training helps the model learn general visual features, while fine-tuning allows the model to focus on specific tasks.

[0044] It is important to note that the encoder of the selected single-task model needs to be able to be layered, that is, to be able to decompose its encoder layer into step-by-step computations, which serves as the basis for architecture search.

[0045] Furthermore, since the auxiliary learning architecture is mainly used to measure and select auxiliary task features that are beneficial to the main task, the model framework is first organized after the single-task model is determined.

[0046] Taking the ViTBase model as an example, image segmentation (patches), linear embedding, positional encoding, and CLS insertion are first packaged into a base operation, that is, the modules before the encoder are encapsulated. Then, the encoder layer is decomposed into step-by-step calculations, and finally, the decoder is connected to different task heads.

[0047] Furthermore, to integrate shallow detail information and deep semantic information from the image, the single-task model employs a neural network structure based on Residual Convolution Units (RCUs) and Fusion Modules (FMMs) to process image data. The RCUs process the input feature maps, reassembling feature maps from different stages into a complete feature map while maintaining the spatial resolution of the feature maps. The FMMs fuse feature maps from different stages and perform upsampling. A Reassemble and a Fusion module are created for each stage in a loop and added to their respective lists to ensure that these modules are correctly invoked during the model's forward propagation.

[0048] Furthermore, neural network head modules for processing normal vector maps and segmented images are designed separately. These modules include a series of convolutional layers, interpolation layers, and activation functions, which serve as decoders to decode the feature maps generated by ViT into the final output.

[0049] Based on the above steps, a search space for feature fusion operations is constructed.

[0050] First, multiple alternative fusion operations are identified, and these alternative fusion operations represent the method of feature fusion.

[0051] Optionally, alternative fusion operations may include feature fusion methods such as splicing, addition, and pooling.

[0052] In the assisted learning architecture search method of the present invention, the alternative fusion operations include neural discrimination dimensionality reduction (NDDR) operations with convolution kernel sizes of 1×1, 3×3, and 5×5.

[0053] Specifically, NDDR (Neural Discriminative Dimensionality Reduction) is a convolutional neural network architecture used for multi-task learning. NDDR first concatenates features of the same spatial resolution from different tasks according to their channel dimensions. Then, it achieves dimensionality reduction through convolution operations, batch normalization, and weight decay. This helps reduce the dimensionality of features while preserving key information, ensuring that the fused features meet the input requirements of subsequent network layers. The effect of neural architecture search is achieved by setting different convolutional kernel sizes.

[0054] In other feasible implementations, alternative fusion operations may also be 7×7 conv NDDR, or disconnection operations.

[0055] After determining the candidate fusion operations, the connection matrix is ​​constructed and initialized based on the number of encoder layers in the single-task model and the number of candidate fusion operations. It can be understood that the connection matrix represents the search range of the auxiliary learning architecture. In other words, this search is to determine the best feature fusion method for each encoder layer of the single-task model among multiple candidate fusion operations, thereby obtaining the final auxiliary learning neural network architecture.

[0056] Before the search, the connection matrix is ​​initialized, that is, the connection weights in the connection matrix are initialized to the same initial value, which is 1 in this embodiment.

[0057] Determine the maximum number of fusion layers and the location of the connection points in the single-task model, so that the input feature position in the i-th layer only receives features from the i-3, i-2, and i-1 layers of another task.

[0058] Step 102: Use Gumbel Softmax to normalize the connection weights in the connection matrix, and use the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks.

[0059] Research has found that during the NAS search process, if the architecture parameters are not constrained or are only limited to a range of 0 to 1, the final architecture parameters will be concentrated on a certain operation, which is not conducive to the model exploring more diverse and richer combinations of operations.

[0060] It is understandable that the architecture parameters represent the parameters of the neural network search architecture, which in this embodiment are the relevant parameters of the connection matrix.

[0061] Therefore, in order to constrain the multivariate parameters and optimize the model convergence results, Gumbel Softmax is used to normalize the connection weights in the connection matrix, and then the processed connection weights are used to perform weighted fusion of single-task features.

[0062] It is important to note that when constraining multivariate parameters, the softmax function is typically used, treating the architecture parameters as the probability distribution corresponding to each operation. However, since the connection matrix constructed in this implementation is initialized to be equal for all parameters, using softmax for normalization constraints can affect the parameter convergence speed. Therefore, the effects of random initialization of architecture parameters, setting upper and lower bounds for parameters (using the `torch.clamp` function), or no constraints on the model convergence results were further explored. Ultimately, Gumbel Softmax was introduced to first introduce noise into the parameters before normalization. By introducing uncertainty, the amplitude of connection weight fluctuations was increased, allowing the model to explore a wider space, resulting in better model convergence.

[0063] Step 103: Apply regularization constraints to the connection weights from the auxiliary task to the main task. When the connection weights from the auxiliary task to the main task converge to near zero during training, the target network architecture is obtained.

[0064] Finally, the optimizer type used by the single-task model is set, the loss function is defined according to the neural network head module of the main task and each auxiliary task, and the model parameters and architecture parameters are updated using gradients from backpropagation.

[0065] Among them, the model parameters are the parameters of the selected single-task model; the architecture parameters represent the parameters of the neural network search architecture.

[0066] like Figure 2 As shown, the training process in this implementation can be understood as initializing bidirectional connections and guiding NAS optimization in the search space, using regularization constraints to make the optimization process converge to an asymmetric network architecture with only unidirectional connections from the main task to the auxiliary task. This allows the connections from the auxiliary task to the main task to be safely removed during the main task inference, thereby achieving multi-task-level performance and single-task inference cost.

[0067] In other words, when the connection weights from the auxiliary task to the main task converge to near zero during training, the model is considered to have been optimized into an asymmetric network architecture with only unidirectional connections from the main task to the auxiliary task. This architecture is then identified as the searched auxiliary task architecture, which is the target network architecture. At this point, the candidate fusion operations corresponding to each encoder layer are determined.

[0068] This invention creates a search space with multiple selectable fusion operations, enabling the model to dynamically adjust the fusion strategy based on different sample features. It uses Gumbel Softmax to model the distribution of discrete weight parameters, and through continuous optimization during training, ensures that the network autonomously selects the most suitable convolution operation in each feature fusion operation. Furthermore, it trains learnable parameters as both model parameters and architecture parameters simultaneously, improving the algorithm's efficiency and flexibility.

[0069] In the assisted learning architecture search method of the present invention, the connection matrix is ​​a three-dimensional connection matrix of N×N×S, where N is the number of encoder layers of the single-task model and S is the number of candidate fusion operations; the first dimension of the connection matrix represents the number of encoder layers where the starting position of feature fusion is located, the second dimension represents the number of encoder layers where the target position of feature fusion is located, and the third dimension represents the fusion operation used at the current connection point.

[0070] Taking the ViTBase model as a single-task model as an example, the three-dimensional connection matrix in this embodiment is explained. A three-dimensional connection matrix of size N×N×S is constructed based on the number of encoder layers N and the number of elements S in the search space of ViTBase, and all its element values ​​are initialized to the same preset value, which is set to 1 in this embodiment.

[0071] The elements in the search space are the alternative fusion operations.

[0072] It is understandable that the number of encoder layers will vary depending on the specific single-task model; the number of candidate fusion operations will also vary depending on the specific candidate fusion operations, and the constructed 3D connection matrix will be adjusted accordingly.

[0073] In the assisted learning architecture search method of this invention, the step of normalizing the connection weights in the connection matrix using Gumbel Softmax specifically includes:

[0074] The connection weights in the connection matrix are scaled after adding Gumbel noise.

[0075] Apply the softmax function to the connection weights in the scaled connection matrix.

[0076] To achieve the purpose of NAS, the most suitable fusion strategy is selected for each connection point, while ensuring the existence of gradients in the weight parameters. In this embodiment of the invention, Gumbel Softmax is used to process the weight parameters.

[0077] Specifically, such as Figure 3 As shown, Gumbel noise is first added to the connection weight parameters of the connection matrix to make it random. Gumbel noise is sampled from the Gumbel distribution, an extremum distribution used to simulate the distribution formed by extrema sampled from other distributions. The formula for Gumbel noise is:

[0078] ;

[0079] In the formula, From uniform distribution Random variables sampled from the middle.

[0080] Then, the noisy weight parameters are scaled and the Softmax function is applied to make the output approximate a one-hot vector while maintaining continuity, thus allowing for gradient calculation. The formula for Softmax is:

[0081] ;

[0082] In the formula, This is a temperature parameter used to control the smoothness of the output. When When the value approaches 0, the output is close to a one-hot vector. As the value approaches infinity, the output becomes nearly uniformly distributed.

[0083] Gumbel Softmax utilizes reparameterization techniques to transfer the non-differentiable sampling process to Gumbel noise. Since Gumbel noise itself does not depend on the original probability distribution, the entire sampling process becomes differentiable. This allows model parameters to be updated via backpropagation, achieving differentiability of discrete variables.

[0084] Based on this, the weight parameters processed by Gumbel Softmax exhibit a one-hot distribution, thus enabling the selection of NAS fusion operations. These parameters are multiplied by the features of the corresponding connection points to achieve feature weighting. The feature fusion process formula in the i-th layer of the model is as follows:

[0085] ;

[0086] In the formula, P i and A i They represent the first i The layer outputs the main task features and auxiliary task features. Here, represents the network architecture parameters. The superscript P indicates that this is a connection from the auxiliary task to the main task, and the subscripts i and j indicate that this parameter is the connection weight from the i-th layer of the auxiliary task to the j-th layer of the main task. 0, 1, and 2 correspond to three determined alternative fusion operations, respectively.

[0087] After performing weighted fusion, normalization, and activation operations, the softmax function is applied to generate a smooth, approximately discrete distribution. The resulting features of this layer are then used as the input to the (i+1)th layer encoder for further forward propagation.

[0088] In the assisted learning architecture search method of the present invention, before the step of constructing and initializing the connection matrix based on the number of encoder layers of the single-task model and the number of candidate fusion operations, the method further includes:

[0089] The encoder of the single-task model is decomposed into a separate encoder layer;

[0090] Extract the input and output features of a single task at each encoder layer.

[0091] To construct the NAS search space, the encoder of the single-task model needs to be split in advance, and the input and output features of each encoder layer obtained from the split need to be determined so that they can be computed when searching the architecture.

[0092] Taking the ViTBase model as an example, we break down its 12 stages of encoder layer into step-by-step calculations. We use loops to extract the input and output features of a single task in each encoder layer (stage), realize the layer-by-layer operation of the data stream, and prepare the conditions for feature fusion between tasks in subsequent steps.

[0093] In the auxiliary learning architecture search method of the present invention, the loss function for regularizing the connection weights from the auxiliary task to the main task is determined based on the loss of the main task, the loss of the auxiliary task, and the regularization term.

[0094] In this implementation, the loss function for architecture search is determined based on the loss of the main task, the loss of the auxiliary task, and the regularization term, as shown below:

[0095] ;

[0096] In the formula, and These correspond to the loss function of the main task header and the loss function of the auxiliary task header, respectively. The added regularization constraint on the connection weights from auxiliary tasks to the main task, where .

[0097] This invention designs a flexible asymmetric network structure that can use different networks in the training and inference phases. It avoids negative transfer between different tasks by sharing soft parameters and learning the independent features of each task. At the same time, it introduces regularization to guide NAS optimization to converge to a connection architecture with only one side from the main task to the auxiliary task, thereby achieving single-task inference cost in the inference phase and further improving the performance of auxiliary learning and architecture search techniques.

[0098] Furthermore, the comparison results between the assisted learning architecture search method of the present invention and the inference results of a single task are shown in Table 1 below.

[0099] Table 1

[0100]

[0101] In the table, task P on the left corresponds to the main task, and task A corresponds to the auxiliary task. The bolded numbers on the right are the optimal metrics. It can be seen that when the selected single-task backbone network is ViTBase, the dataset used is NYU-v2, and the two tasks are set as image segmentation and planar phase prediction respectively, the running results constructed using the algorithm in this embodiment show better performance for both the main task and the auxiliary task.

[0102] The auxiliary learning architecture search device provided by the present invention is described below. The auxiliary learning architecture search device described below can be referred to in correspondence with the auxiliary learning architecture search method described above.

[0103] like Figure 4 As shown, the auxiliary learning architecture search device provided by the present invention includes a construction module 401, a fusion module 402, and a determination module 403:

[0104] The construction module 401 is used to construct and initialize the connection matrix based on the number of encoder layers of the single-task model and the number of alternative fusion operations, wherein the single-task model is a model pre-determined for executing the main task and several auxiliary tasks.

[0105] Understandably, as a form of multi-task learning, an efficient auxiliary learning neural network architecture must simultaneously consider both the feature-sharing part and the task-specific part, and be able to reasonably measure and select auxiliary task features that are beneficial to the main task. Therefore, designing a high-performance neural network architecture becomes very important.

[0106] Traditional neural network architecture design relies on expert experience and extensive experimentation, a process that is both time-consuming and lacks systematic approach. In recent years, however, the development of Neural Architecture Search (NAS) technology has made automated network architecture design possible.

[0107] NAS (Network Attached Search) uses algorithms to automatically explore different network architectures in order to discover the most effective model for the target task. NAS defines a search space containing multiple possible network architectures, and then uses intelligent search strategies to search within the space and evaluate the performance of different architectures for iterative optimization.

[0108] Therefore, the first step in developing a search-assisted learning architecture is to determine the search space of the NAS based on the primary and secondary tasks.

[0109] Optionally, a main task can correspond to one auxiliary task or multiple auxiliary tasks.

[0110] Optionally, a single-task model for the task to be performed can be determined based on the categories of the main task and auxiliary tasks, and a search space can be constructed based on the selected single-task model.

[0111] In this embodiment, semantic segmentation of an image is taken as the primary task and plane normal prediction is taken as the auxiliary task for illustration. It can be understood that since both the primary and auxiliary tasks selected in this embodiment are image processing tasks, ViTBase is selected as the backbone network for a single image task (primary task or auxiliary task), that is, ViTBase is determined as a single-task model.

[0112] ViTBase employs a Transformer architecture. The input image is first segmented into multiple patches, each treated as a "token." These patches are embedded into a high-dimensional space and then fed into the Transformer model. The ViTBase model utilizes a self-attention mechanism to capture the relationships between different regions in the image. This model is typically pre-trained on large-scale image datasets and then fine-tuned for specific tasks. Pre-training helps the model learn general visual features, while fine-tuning allows the model to focus on specific tasks.

[0113] It is important to note that the encoder of the selected single-task model needs to be able to be layered, that is, to be able to decompose its encoder layer into step-by-step computations, which serves as the basis for architecture search.

[0114] Furthermore, since the auxiliary learning architecture is mainly used to measure and select auxiliary task features that are beneficial to the main task, the model framework is first organized after the single-task model is determined.

[0115] Taking the ViTBase model as an example, image segmentation (patches), linear embedding, positional encoding, and CLS insertion are first packaged into a base operation, that is, the modules before the encoder are encapsulated. Then, the encoder layer is decomposed into step-by-step calculations, and finally, the decoder is connected to different task heads.

[0116] Furthermore, to integrate shallow detail information and deep semantic information from the image, the single-task model employs a neural network structure based on Residual Convolution Units (RCUs) and Fusion Modules (FMMs) to process image data. The RCUs process the input feature maps, reassembling feature maps from different stages into a complete feature map while maintaining the spatial resolution of the feature maps. The FMMs fuse feature maps from different stages and perform upsampling. A Reassemble and a Fusion module are created for each stage in a loop and added to their respective lists to ensure that these modules are correctly invoked during the model's forward propagation.

[0117] Furthermore, neural network head modules for processing normal vector maps and segmented images are designed separately. These modules include a series of convolutional layers, interpolation layers, and activation functions, which serve as decoders to decode the feature maps generated by ViT into the final output.

[0118] Based on the above steps, a search space for feature fusion operations is constructed.

[0119] First, multiple alternative fusion operations are identified, and these alternative fusion operations represent the method of feature fusion.

[0120] Optionally, alternative fusion operations may include feature fusion methods such as splicing, addition, and pooling.

[0121] After determining the candidate fusion operations, the connection matrix is ​​constructed and initialized based on the number of encoder layers in the single-task model and the number of candidate fusion operations. It can be understood that the connection matrix represents the search range of the auxiliary learning architecture. In other words, this search is to determine the best feature fusion method for each encoder layer of the single-task model among multiple candidate fusion operations, thereby obtaining the final auxiliary learning neural network architecture.

[0122] The fusion module 402 is used to normalize the connection weights in the connection matrix using Gumbel Softmax, and to use the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks.

[0123] Research has found that during the NAS search process, if the architecture parameters are not constrained or are only limited to a range of 0 to 1, the final architecture parameters will be concentrated on a certain operation, which is not conducive to the model exploring more diverse and richer combinations of operations.

[0124] It is understandable that the architecture parameters represent the parameters of the neural network search architecture, which in this embodiment are the relevant parameters of the connection matrix.

[0125] Therefore, in order to constrain the multivariate parameters and optimize the model convergence results, Gumbel Softmax is used to normalize the connection weights in the connection matrix, and then the processed connection weights are used to perform weighted fusion of single-task features.

[0126] It is important to note that when constraining multivariate parameters, the softmax function is typically used, treating the architecture parameters as the probability distribution corresponding to each operation. However, since the connection matrix constructed in this implementation is initialized to be equal for all parameters, using softmax for normalization constraints can affect the parameter convergence speed. Therefore, the effects of random initialization of architecture parameters, setting upper and lower bounds for parameters (using the `torch.clamp` function), or no constraints on the model convergence results were further explored. Ultimately, Gumbel Softmax was introduced to first introduce noise into the parameters before normalization. By introducing uncertainty, the amplitude of connection weight fluctuations was increased, allowing the model to explore a wider space, resulting in better model convergence.

[0127] The determination module 403 is used to perform regularization constraints on the connection weights from the auxiliary task to the main task. When the connection weights from the auxiliary task to the main task converge to close to zero during training, the target network architecture is obtained.

[0128] Finally, the optimizer type used by the single-task model is set, the loss function is defined according to the neural network head module of the main task and each auxiliary task, and the model parameters and architecture parameters are updated using gradients from backpropagation.

[0129] Among them, the model parameters are the parameters of the selected single-task model; the architecture parameters represent the parameters of the neural network search architecture.

[0130] like Figure 2 As shown, the training process in this implementation can be understood as initializing bidirectional connections and guiding NAS optimization in the search space, using regularization constraints to make the optimization process converge to an asymmetric network architecture with only unidirectional connections from the main task to the auxiliary task. This allows the connections from the auxiliary task to the main task to be safely removed during the main task inference, thereby achieving multi-task-level performance and single-task inference cost.

[0131] In other words, when the connection weights from the auxiliary task to the main task converge to near zero during training, the model is considered to have been optimized into an asymmetric network architecture with only unidirectional connections from the main task to the auxiliary task. This architecture is then identified as the searched auxiliary task architecture, which is the target network architecture. At this point, the candidate fusion operations corresponding to each encoder layer are determined.

[0132] This invention creates a search space with multiple selectable fusion operations, enabling the model to dynamically adjust the fusion strategy based on different sample features. It uses Gumbel Softmax to model the distribution of discrete weight parameters, and through continuous optimization during training, ensures that the network autonomously selects the most suitable convolution operation in each feature fusion operation. Furthermore, it trains learnable parameters as both model parameters and architecture parameters simultaneously, improving the algorithm's efficiency and flexibility.

[0133] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include a processor 510, a communication interface 520, a memory 530, and a communication bus 540, wherein the processor 510, the communication interface 520, and the memory 530 communicate with each other through the communication bus 540. The processor 510 can call logical instructions in the memory 530 to execute an auxiliary learning architecture search method. This method includes: constructing and initializing a connection matrix based on the number of encoder layers in a single-task model and the number of candidate fusion operations, wherein the single-task model is a model pre-determined for executing a main task and several auxiliary tasks; normalizing the connection weights in the connection matrix using Gumbel Softmax, and using the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks; applying regularization constraints to the connection weights from the auxiliary tasks to the main task, and obtaining the target network architecture when the connection weights from the auxiliary tasks to the main task converge to near zero during training.

[0134] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0135] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the auxiliary learning architecture search method provided by the above methods. The method includes: constructing and initializing a connection matrix based on the number of encoder layers of a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and several auxiliary tasks; normalizing the connection weights in the connection matrix using Gumbel Softmax, and using the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks; applying regularization constraints to the connection weights from the auxiliary tasks to the main task, and obtaining the target network architecture when the connection weights from the auxiliary tasks to the main task converge to near zero during training.

[0136] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the auxiliary learning architecture search method provided by the above methods. This method includes: constructing and initializing a connection matrix based on the number of encoder layers in a single-task model and the number of candidate fusion operations, wherein the single-task model is a model predetermined for performing a main task and several auxiliary tasks; normalizing the connection weights in the connection matrix using GumbelSoftmax, and using the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks; applying regularization constraints to the connection weights from the auxiliary tasks to the main task, and obtaining the target network architecture when the connection weights from the auxiliary tasks to the main task converge to near zero during training.

[0137] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0138] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0139] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for searching an assisted learning architecture, characterized in that, include: The connection matrix is ​​constructed and initialized based on the number of encoder layers in the single-task model and the number of candidate fusion operations, wherein the single-task model is a model pre-determined for executing the main task and several auxiliary tasks; The connection weights in the connection matrix are normalized using Gumbel Softmax, and the processed connection weights are used to perform weighted fusion of features between the main task and several auxiliary tasks. The connection weights from the auxiliary task to the main task are subject to regularization constraints. The target network architecture is obtained when the connection weights from the auxiliary task to the main task converge to near zero during training. In this model, semantic segmentation of the image is taken as the main task and plane normal prediction is taken as the auxiliary task. The single-task model adopts ViTBase and uses the image as the input of the single-task model. The connection matrix is ​​a three-dimensional connection matrix of N×N×S, where N is the number of encoder layers of the single-task model and S is the number of candidate fusion operations; the first dimension of the connection matrix represents the number of encoder layers where the starting position of feature fusion is located, the second dimension represents the number of encoder layers where the target position of feature fusion is located, and the third dimension represents the fusion operation used at the current connection point.

2. The assisted learning architecture search method according to claim 1, characterized in that, The alternative fusion operations include neural discrimination dimensionality reduction (NDDR) operations with kernel sizes of 1×1, 3×3, and 5×5.

3. The assisted learning architecture search method according to claim 1 or 2, characterized in that, The step of normalizing the connection weights in the connection matrix using GumbelSoftmax specifically includes: The connection weights in the connection matrix are scaled after adding Gumbel noise. Apply the softmax function to the connection weights in the scaled connection matrix.

4. The assisted learning architecture search method according to claim 1 or 2, characterized in that, Before the step of constructing and initializing the connection matrix based on the number of encoder layers in the single-task model and the number of candidate fusion operations, the method further includes: The encoder of the single-task model is decomposed into a separate encoder layer; Extract the input and output features of a single task at each encoder layer.

5. The assisted learning architecture search method according to claim 1 or 2, characterized in that, The loss function that applies regularization constraints to the connection weights from the auxiliary task to the main task is determined based on the loss of the main task, the loss of the auxiliary task, and the regularization term.

6. A learning-assisted architecture search device, characterized in that, include: A construction module is used to construct and initialize a connection matrix based on the number of encoder layers in a single-task model and the number of alternative fusion operations, wherein the single-task model is a model pre-determined for executing the main task and several auxiliary tasks. The fusion module is used to normalize the connection weights in the connection matrix using Gumbel Softmax, and then use the processed connection weights to perform weighted fusion of features between the main task and several auxiliary tasks. The determination module is used to perform regularization constraints on the connection weights from the auxiliary task to the main task, and obtain the target network architecture when the connection weights from the auxiliary task to the main task converge to close to zero during training. In this model, semantic segmentation of the image is taken as the main task and plane normal prediction is taken as the auxiliary task. The single-task model adopts ViTBase and uses the image as the input of the single-task model. The connection matrix is ​​a three-dimensional connection matrix of N×N×S, where N is the number of encoder layers of the single-task model and S is the number of candidate fusion operations; the first dimension of the connection matrix represents the number of encoder layers where the starting position of feature fusion is located, the second dimension represents the number of encoder layers where the target position of feature fusion is located, and the third dimension represents the fusion operation used at the current connection point.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the assisted learning architecture search method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the assisted learning architecture search method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the assisted learning architecture search method as described in any one of claims 1 to 5.

Images