Self-attention mechanism representation network search method, device and equipment
By constructing a self-attention mechanism to represent the network search space and supernetwork, the optimal architecture is designed automatically, solving the problems of low design efficiency and applicability of the self-attention mechanism-represented network, and realizing efficient network architecture search.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-07-19
- Publication Date
- 2026-03-24
AI Technical Summary
In existing technologies, the design efficiency of self-attention mechanism representation networks is low, they rely on expert experience and are prone to introducing design biases. The search space and search algorithm are not suitable for self-attention mechanism representation networks, and they cannot solve their high flexibility and heterogeneity problems.
A self-attention mechanism is constructed to represent the network search space, which contains multiple subnetworks. The optimal architecture is searched through a supernetwork, pruning is performed using constraint policies, and the supernetwork is optimized through training data to achieve automated design.
It improves the design efficiency of self-attention mechanism representation networks, reduces human and technical costs, and enhances the representation ability and search efficiency of deep models.
Smart Images
Figure CN113673673B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and more specifically, to a self-attention mechanism characterization method, apparatus, and device for network search. Background Technology
[0002] Self-attention mechanisms are currently widely used in natural language processing and graph representation learning tasks. Deep neural networks built using self-attention mechanisms can greatly improve the training speed and representation capabilities of neural networks. Existing techniques involve manually designing suitable self-attention mechanisms to represent networks for each task and dataset in the natural language processing and graph representation learning domains. Neural network architecture search improves the efficiency and effectiveness of deep network model design by automating the design of deep networks. This type of technique generally includes two parts: a search space and a search algorithm. The search space contains various deep neural network architectures, and the search algorithms mainly include sub-algorithms based on reinforcement learning, evolutionary algorithms, and differentiable algorithms.
[0003] In existing technologies, the design of self-attention mechanisms mainly relies on expert experience and manual debugging, which is inefficient, costly in terms of manpower and technology, and prone to introducing design biases and suboptimal designs. Existing neural network search architectures do not specifically design representation networks for self-attention mechanisms, and their search spaces and algorithms cannot be directly applied to these networks, failing to address the issues of high flexibility and heterogeneity inherent in self-attention mechanism representation networks. Summary of the Invention
[0004] This application provides a method, apparatus, and device for searching self-attention mechanism representation networks, aiming to accelerate the design process of self-attention mechanism representation networks and improve the design efficiency and effectiveness of deep models.
[0005] A first aspect of this application provides a self-attention mechanism representation network search method, the method comprising:
[0006] A self-attention mechanism representation network search space is constructed based on self-attention mechanism representation networks with various architectures, and the self-attention mechanism representation network search space contains multiple sub-networks;
[0007] A supernetwork is constructed using the multiple subnetworks;
[0008] The data to be analyzed is input into the hypernetwork, and the corresponding self-attention mechanism representation network is searched from the self-attention mechanism representation network search space through the hypernetwork.
[0009] Optionally, the method further includes:
[0010] Collect training data for the target task, and put the training data into a set to obtain the training dataset;
[0011] The training dataset is input into the hypernetwork to train the hypernetwork and obtain a trained hypernetwork.
[0012] Optionally, a search space for a self-attention mechanism representation network is constructed based on self-attention mechanism representation networks with various architectures, including:
[0013] Based on the structural features of the self-attention mechanism characterization network of the various architectures, multiple network fusion layers corresponding to each self-attention mechanism characterization network are obtained;
[0014] The multiple network fusion layers are connected to obtain the multiple sub-networks;
[0015] The self-attention mechanism is constructed using the multiple sub-networks to characterize the network search space.
[0016] Optionally, the method further includes:
[0017] The self-attention mechanism is used to prune the network search space.
[0018] Optionally, the constraint strategies include: a trunk constraint strategy, a key-value binding constraint strategy, and a zero-operation constraint strategy.
[0019] Optionally, a supernetwork is constructed through the plurality of subnetworks, including:
[0020] Parameters with the same context in the multiple sub-networks are set as parameters at the same location in the supernetwork;
[0021] Parameters with different contexts in the multiple sub-networks are set as parameters at different positions in the supernetwork.
[0022] Optionally, the method further includes:
[0023] The corresponding self-attention mechanism representation network is trained to obtain a trained self-attention mechanism representation network.
[0024] Optionally, based on the structural features of the self-attention mechanism characterization networks of the various architectures, multiple network fusion layers corresponding to each self-attention mechanism characterization network are obtained, including:
[0025] Based on the self-attention mechanism representation network of the various architectures, the self-attention mechanism fusion layer corresponding to each self-attention mechanism representation network is obtained;
[0026] Based on the layers other than the self-attention layer in the self-attention mechanism representation network of the various architectures, an additive fusion layer corresponding to each self-attention mechanism representation network is obtained.
[0027] A second aspect of this application provides a self-attention mechanism representation network generation apparatus, the apparatus comprising:
[0028] The search space construction module is used to construct a self-attention mechanism representation network search space based on self-attention mechanism representation networks of various architectures. The self-attention mechanism representation network search space contains multiple sub-networks.
[0029] A hypernetwork construction module is used to construct a hypernetwork from the multiple subnetworks;
[0030] The network acquisition module is used to input the data to be analyzed into the supernetwork, and search for the corresponding self-attention mechanism representation network from the self-attention mechanism representation network search space through the supernetwork.
[0031] Optionally, the device further includes:
[0032] The training dataset generation module is used to collect training data for the target task, put the training data into a set, and obtain the training dataset.
[0033] The hypernetwork training module is used to input the training dataset into the hypernetwork, train the hypernetwork, and obtain a trained hypernetwork.
[0034] Optionally, the search space construction module includes:
[0035] The network fusion layer obtaining sub-module is used to obtain multiple network fusion layers corresponding to each self-attention mechanism characterizing network based on the structural features of the network characterized by the self-attention mechanism of the various architectures.
[0036] A sub-network generation sub-module is used to connect the multiple network fusion layers to obtain the multiple sub-networks;
[0037] The search space construction submodule is used to construct the self-attention mechanism characterization network search space through the multiple sub-networks.
[0038] Optionally, the search space construction module further includes:
[0039] The spatial pruning submodule is used to prune the search space of the self-attention mechanism characterization network using a constraint strategy.
[0040] Optionally, the constraint strategies include: a trunk constraint strategy, a key-value binding constraint strategy, and a zero-operation constraint strategy.
[0041] Optionally, the hypernetwork building module includes:
[0042] The first parameter setting submodule is used to set parameters with the same context in the multiple sub-networks to parameters at the same position in the supernetwork;
[0043] The second parameter setting submodule is used to set parameters with different contexts in the multiple subnetworks to parameters at different positions in the supernetwork.
[0044] Optionally, the device further includes:
[0045] The corresponding self-attention mechanism representation network is trained to obtain a trained self-attention mechanism representation network.
[0046] Optionally, the network fusion layer submodule includes:
[0047] The self-attention layer acquisition submodule is used to obtain the self-attention mechanism fusion layer corresponding to each self-attention mechanism representation network based on the self-attention mechanism representation network of the various architectures.
[0048] The additive fusion layer acquisition submodule is used to obtain the additive fusion layer corresponding to each self-attention mechanism representation network based on the other layers in the self-attention mechanism representation network of the various architectures, excluding the self-attention layer.
[0049] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the method described in the first aspect of this application.
[0050] The self-attention mechanism representation network search method provided in this application constructs a self-attention mechanism representation network search space based on various self-attention mechanism representation network architectures. This search space contains multiple sub-networks. The data to be analyzed is input into the super-network, and the corresponding self-attention mechanism representation network is searched for within the search space using the super-network. This application constructs a self-attention mechanism representation network search space by using various self-attention mechanism representation networks as sub-networks, and then constructs a super-network using the multiple sub-networks within this search space. Using this super-network, the self-attention mechanism representation network corresponding to the dataset to be analyzed can be searched within the search space. This achieves automated search for the optimal self-attention mechanism representation network architecture, saves time in constructing self-attention mechanism representation networks, and improves the efficiency of building deep neural networks. Attached Figure Description
[0051] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0052] Figure 1 This is a flowchart of a self-attention mechanism characterization network search method proposed in an embodiment of this application;
[0053] Figure 2 This is a schematic diagram of a sub-network structure proposed in an embodiment of this application;
[0054] Figure 3 This is a schematic diagram of a self-attention mechanism characterizing a network search device according to an embodiment of this application. Detailed Implementation
[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0056] refer to Figure 1 , Figure 1 This is a flowchart illustrating a self-attention mechanism-based network search method proposed in an embodiment of this application. Figure 1 As shown, the method includes the following steps:
[0057] S11: Construct a self-attention mechanism representation network search space based on self-attention mechanism representation networks with multiple architectures, wherein the self-attention mechanism representation network search space contains multiple sub-networks.
[0058] In this embodiment, the self-attention mechanism representation network is a deep neural network built on the self-attention mechanism. However, different self-attention mechanism representation networks have different architectures. Therefore, it is necessary to construct an attention mechanism guarantee network search space based on the self-attention mechanism representation networks with various architectures as sub-networks. The self-attention mechanism representation network search space is a set of multiple self-attention mechanism representation networks with different architectures.
[0059] In this embodiment, the specific steps for constructing a self-attention mechanism representation network search space based on various architectures of self-attention mechanism representation networks include:
[0060] S11-1: Based on the structural features of the self-attention mechanism characterization network of the various architectures, obtain multiple network fusion layers corresponding to each self-attention mechanism characterization network.
[0061] In this embodiment, the self-attention mechanism representation networks of various architectures are pre-designed self-attention mechanism representation networks. By observing the structural features of each self-attention mechanism representation network, each self-attention mechanism representation network can be abstracted into a unified standard form, which makes it convenient to put these self-attention mechanism representation networks of various architectures into the same space in the future.
[0062] In this embodiment, each self-attention mechanism-represented network is abstracted into the concepts of layers and connections, and the construction task of deep neural networks is modeled as a multi-stacked fusion layer selection problem. A network fusion layer is a layer structure abstracted from the structure of the self-attention mechanism-represented network. Network fusion layers are divided into two types: additive fusion layers (additive layers) and self-attention mechanism fusion layers (attention layers). The specific steps for obtaining multiple network fusion layers corresponding to each self-attention mechanism-represented network, based on the structural characteristics of the various self-attention mechanism-represented networks, are as follows:
[0063] S11-1-1: Based on the self-attention layer in the self-attention mechanism representation network of the various architectures, obtain the self-attention mechanism fusion layer corresponding to each self-attention mechanism representation network.
[0064] In this embodiment, the self-attention mechanism representation network contains several self-attention layers that process the input vector using the self-attention mechanism. Different weights are assigned to different parts of the processed vector. Based on the self-attention layers in each self-attention mechanism representation network, these layers are processed and abstracted into a standard expression, thus obtaining the corresponding self-attention mechanism fusion layer, which is used to construct the self-attention mechanism search space.
[0065] In this embodiment, the attention layer accepts three inputs and performs an attention information fusion operation. The three inputs are a query input, a key input, and a value input, and the output is a vector processed by the self-attention mechanism. Through this self-attention fusion layer, the input data can be processed by the self-attention mechanism.
[0066] S11-1-2: Based on the other layers in the self-attention mechanism representation network of the various architectures, excluding the self-attention layer, obtain the additive fusion layer corresponding to each self-attention mechanism representation network.
[0067] In this embodiment, the additive layer is obtained by characterizing the non-attention mechanism layer in the network based on the self-attention mechanism, and mainly includes the feature extraction layer, etc.
[0068] In this embodiment, the additive layer receives two inputs, performs summation on these two operations, and outputs a vector after summation.
[0069] For example, the additive layer receives input 1 and input 2, corresponding to operation 1 and operation 2. Operation 1 can be a feature extraction operation performed by a feature extraction network, and operation 2 can be a feature extraction operation performed by another feature extraction network. After receiving operation 1 and operation 2, the feature extraction results of operation 1 and operation 2 are fused.
[0070] S11-2: Connect the multiple network fusion layers to obtain the multiple sub-networks.
[0071] In this embodiment, a subnetwork is a network structure composed of multiple network fusion layers. There are multiple subnetworks in a search space, and each subnetwork has a different architecture. Multiple subnetworks can be obtained by connecting multiple network fusion layers.
[0072] In this embodiment, the structural features of the network are characterized by the original self-attention mechanism, the connection methods of each structure are abstracted, and the network fusion layer is connected according to the connection methods of each structure to obtain multiple sub-networks.
[0073] For example, the structure of a subnetwork could be that layer 1 is an attention layer, layers 2 and 3 are additive layers, and layers 4 and 5 are attention layers.
[0074] S11-3: The self-attention mechanism is constructed through the multiple sub-networks to characterize the network search space.
[0075] In this embodiment, by adding multiple sub-networks into the same space, the self-attention mechanism representation network search space is constructed.
[0076] S12: Construct a supernetwork through the multiple subnetworks.
[0077] In this embodiment, the hypernetwork is a network search model that can search for several subnetworks from multiple subnetworks in the network search space when data is received, sort them, and place the subnetwork with the highest score at the top, which is the most suitable subnetwork for processing the data.
[0078] In this embodiment, by constructing all subnetworks in the search space as a supernetwork, all subnetworks in the search space can be jointly optimized and updated, which greatly improves the efficiency of the search model's search architecture.
[0079] In this embodiment, the specific steps for constructing a supernetwork using the multiple subnetworks are as follows:
[0080] S12-1: Set the parameters with the same context in the multiple sub-networks to the parameters at the same position in the supernetwork.
[0081] S12-2: Set the parameters with different contexts in the multiple sub-networks to parameters at different positions in the supernetwork.
[0082] In this embodiment, the context of a parameter refers to the category of the fusion layer it is connected to. Setting parameters with the same context in multiple sub-networks to the same location means setting parameters from different sub-networks with the same fusion layer category and connection method to the same location for parameter sharing. Conversely, different contexts indicate different network fusion layer categories or connection methods within the sub-networks, and these are set as parameters in different locations within the supernetwork.
[0083] For example, in the three subnetworks in the search space, the first layer of subnetwork 1 is an attention layer and the second layer is an additive layer; the first layer of subnetwork 2 is an additive layer and the second layer is an attention layer; and the first layer of subnetwork 3 is an attention layer and the second layer is an additive layer. Then, the first and second layers of subnetwork 1 and subnetwork 3 are set to the same parameters.
[0084] In the search space of self-attention representation networks, there are multiple choices for fusion layers, each with different functions and required connection types, resulting in spatial heterogeneity. Directly constructing a supernetwork for optimization fails to consider this heterogeneity, leading to parameter homogenization and impacting search performance. This embodiment employs a context-dependent supernetwork construction method. Through this method, parameters are shared only between subnetworks with the same fusion layer position and connection method, significantly alleviating the model homogenization problem in the supernetwork model and improving the performance of searching subnetworks.
[0085] S13: Input the data to be analyzed into the supernetwork, and search for the corresponding self-attention mechanism representation network from the self-attention mechanism representation network search space through the supernetwork.
[0086] In this embodiment, the data to be analyzed is the data that needs to be processed. The target task of the data to be analyzed is the same as that of the dataset used to train the supernetwork. After the data to be analyzed is input into the supernetwork, the supernetwork calculates the data to be analyzed and obtains the top N self-attention mechanism representation networks with the highest scores. The self-attention mechanism representation network with the highest score is the most suitable self-attention mechanism representation network for processing the data to be analyzed, that is, the self-attention mechanism representation network corresponding to the data to be analyzed.
[0087] For example, the dataset of images to be classified is input into a hypernetwork trained on a dataset for image classification. This hypernetwork searches the search space for the best self-attention mechanism representation network to classify the images to be classified.
[0088] In another embodiment of this application, after the supernetwork is established, it is first trained to obtain a trained supernetwork. The specific steps are as follows:
[0089] S21: Collect training data for the target task, and put the training data into a set to obtain a training dataset.
[0090] S22: Input the training dataset into the supernetwork, train the supernetwork, and obtain the trained supernetwork.
[0091] In this embodiment, after the hypernetwork is built, it needs to be trained to optimize the parameters of the hypernetwork. Different training data need to be collected for different tasks. Data for different target tasks are put into different datasets to obtain multiple training datasets for different types of tasks.
[0092] After obtaining the training dataset, it is input into the hypernetwork for training. The hypernetwork generates TOPN self-attention mechanism representation networks corresponding to the data in the training dataset. This training process continuously optimizes the hypernetwork parameters. When the hypernetwork parameters are optimal, the search performance for self-attention mechanism representation networks in the search space also reaches its optimal level. A hypernetwork is trained for each task-specific dataset. This hypernetwork can select the optimal self-attention mechanism representation network for the target task. The parameters of the hypernetwork trained on datasets for different tasks are different.
[0093] For example, if the target task is semantic recognition, the training dataset corresponding to the semantic recognition task is input into the hypernetwork, and the hypernetwork is trained to obtain a trained hypernetwork. When receiving a dataset for the semantic recognition task, the trained hypernetwork can quickly find the top N hypernetworks with the best recognition performance for the semantic recognition task in the search space. The operating principle is the same for other tasks.
[0094] In another embodiment of this application, after the search space is established, it is also necessary to prune the space using a constraint strategy, specifically including:
[0095] S23: Use a constraint strategy to prune the search space of the self-attention mechanism characterization network.
[0096] In this embodiment, the constraint strategy is a normalization requirement imposed on the network in the space. The network in the space must meet these constraint strategies to operate normally. Setting the network in the search space according to the constraint strategy realizes the pruning of the search space.
[0097] S24: The constraint strategies include: trunk constraint strategy, key-value binding constraint strategy and zero-operation constraint strategy.
[0098] In this embodiment, the backbone constraint strategy means that all layers have an input from the output of the previous layer. This type of inter-layer connection is called a backbone connection. For the attention layer, the input is fixed as the query input, and for the additive layer, it is any input.
[0099] For example, in a subnetwork within the search space, if the first layer is an additive layer, the second layer is an attention layer, and the third layer is an additive layer, then the output of the first layer is the query input of the second layer, and the output of the second layer can be any input of the third layer.
[0100] The key-value binding constraint strategy means that the key input and value input of the attention layer come from the output of the same layer. This ensures that the key and value come from the same layer, reduces the optimization difficulty of the model, and allows the self-attention mechanism to play its role and become meaningful.
[0101] The zero-operation constraint strategy forces all connections in the backbone and attention layers to have non-zero operations. This is because if the operations on a connection are zero, the output will definitely be zero, resulting in an invalid deep neural network. This strategy avoids the generation of invalid deep neural networks.
[0102] The three constraint strategies described above reduce the complexity of the search space from O(n)! 4 Reduced to O(n!) 2 This significantly reduces algorithm complexity and makes searching much easier.
[0103] In another embodiment of this application, after finding the corresponding self-attention mechanism representation network, the following steps are also required:
[0104] S31: Train the corresponding self-attention mechanism representation network to obtain a trained self-attention mechanism representation network.
[0105] In this embodiment, after obtaining the corresponding self-attention mechanism representation network from the search space, the network is trained using the corresponding dataset, and the parameters of the self-attention mechanism representation network are adjusted to obtain the trained self-attention mechanism representation network.
[0106] This application embodiment automates the design of self-attention mechanism representation networks in deep neural networks for the first time. It can automatically generate the optimal self-attention mechanism representation network from the network search space, thereby greatly improving the expressive power of deep neural networks. At the same time, it greatly reduces the time, resources, and human and technical costs of designing neural networks, improving the efficiency and effectiveness of the design process.
[0107] The embodiments of this application will now be described in conjunction with the accompanying drawings:
[0108] like Figure 2 As shown, Figure 2 This is a schematic diagram of a sub-network structure proposed in an embodiment of this application, as shown below. Figure 2 The following is stated:
[0109] The different logos in the image are distinguished by varying shades of black. From Figure 2 As shown on the left, the sub-network mainly consists of two types of network fusion layers: additive fusion layers and self-attention mechanism fusion layers. The additive fusion layer receives two inputs, fuses operation 1 and operation 2, and obtains one output. The self-attention mechanism fusion layer receives a query input, a key input, and a value input, and obtains one output.
[0110] Figure 2 The middle section also illustrates the structure of the sub-networks. In the diagram, layers 1, 4, and 5 are attention layers, while layers 2 and 3 are additive layers. There are three connections from the input to layer 1: a query connection, a key connection, and a value connection. Layer 2 receives three inputs; layer 2 receives the output and input from layer 1; layer 3 receives the output and input from layer 2; layer 4 receives the output from layer 3 and the key and value passed from layer 2; and layer 5 receives the output from layer 4 and the key and value passed from layer 3.
[0111] from Figure 2 As shown on the right, layers 2, 4, and 6 of subnetwork 1 (above) are additive layers, and layers 2 and 6 of subnetwork 2 (below) are additive layers, with layer 4 being an attention layer. Therefore, since layers 2 and 6 of both subnetwork 1 and subnetwork 2 are additive layers, the data between layers 2 and 6 in subnetwork 1 and subnetwork 2 is set to parameters at the same location, meaning they are parameter-shared. However, because layer 4 of subnetwork 1 is an additive layer and layer 4 of subnetwork 2 is an attention layer, the parameters between layers 2 and 4, and between layers 4 and 6, are set to parameters at different locations, meaning they are not shared.
[0112] Based on the same inventive concept, one embodiment of this application provides a self-attention mechanism characterization network search device. (Reference) Figure 3 , Figure 3 This is a schematic diagram illustrating the self-attention mechanism characterizing a network search device 300 according to an embodiment of this application. Figure 3 As shown, the device includes:
[0113] Search space construction module 301 is used to construct a self-attention mechanism representation network search space based on self-attention mechanism representation networks of various architectures, wherein the self-attention mechanism representation network search space contains multiple sub-networks.
[0114] Hypernetwork construction module 302 is used to construct a hypernetwork through the plurality of subnetworks;
[0115] The network acquisition module 303 is used to input the data to be analyzed into the supernetwork and search for the corresponding self-attention mechanism representation network from the self-attention mechanism representation network search space through the supernetwork.
[0116] Optionally, the device further includes:
[0117] The training dataset generation module is used to collect training data for the target task, put the training data into a set, and obtain the training dataset.
[0118] The hypernetwork training module is used to input the training dataset into the hypernetwork, train the hypernetwork, and obtain a trained hypernetwork.
[0119] Optionally, the search space construction module includes:
[0120] The network fusion layer obtaining sub-module is used to obtain multiple network fusion layers corresponding to each self-attention mechanism characterizing network based on the structural features of the network characterized by the self-attention mechanism of the various architectures.
[0121] A sub-network generation sub-module is used to connect the multiple network fusion layers to obtain the multiple sub-networks;
[0122] The search space construction submodule is used to construct the self-attention mechanism characterization network search space through the multiple sub-networks.
[0123] Optionally, the search space construction module further includes:
[0124] The spatial pruning submodule is used to prune the search space of the self-attention mechanism characterization network using a constraint strategy.
[0125] Optionally, the constraint strategies include: a trunk constraint strategy, a key-value binding constraint strategy, and a zero-operation constraint strategy.
[0126] Optionally, the hypernetwork building module includes:
[0127] The first parameter setting submodule is used to set parameters with the same context in the multiple sub-networks to parameters at the same position in the supernetwork;
[0128] The second parameter setting submodule is used to set parameters with different contexts in the multiple subnetworks to parameters at different positions in the supernetwork.
[0129] Optionally, the device further includes:
[0130] The corresponding self-attention mechanism representation network is trained to obtain a trained self-attention mechanism representation network.
[0131] Optionally, the network fusion layer submodule includes:
[0132] The self-attention layer acquisition submodule is used to obtain the self-attention mechanism fusion layer corresponding to each self-attention mechanism representation network based on the self-attention mechanism representation network of the various architectures.
[0133] The additive fusion layer acquisition submodule is used to obtain the additive fusion layer corresponding to each self-attention mechanism representation network based on the other layers in the self-attention mechanism representation network of the various architectures, excluding the self-attention layer.
[0134] Based on the same inventive concept, another embodiment of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the steps in the self-attention mechanism characterization network search method described in any of the above embodiments of this application.
[0135] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.
[0136] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0137] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0138] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0139] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0140] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0141] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.
[0142] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.
[0143] The foregoing has provided a detailed description of the self-attention mechanism characterization network search method, apparatus, and device provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.
Claims
1. A self-attention mechanism for representing network search, characterized in that, The method includes: Constructing a self-attention mechanism representation network search space based on self-attention mechanism representation networks of various architectures includes: constructing a self-attention mechanism representation network search space based on the structural characteristics of the self-attention mechanism representation networks of various architectures, wherein the self-attention mechanism representation network search space contains multiple sub-networks; A supernetwork is constructed using the multiple subnetworks. Specifically, parameters with the same context in the multiple subnetworks are set as parameters at the same position in the supernetwork; parameters with different contexts in the multiple subnetworks are set as parameters at different positions in the supernetwork. Collect training data for the semantic recognition task, put the training data into a set, and obtain the training dataset corresponding to the semantic recognition task. The training dataset corresponding to the semantic recognition task is input into the hypernetwork to train the hypernetwork and obtain a trained hypernetwork for the semantic recognition task. During the training process of the hypernetwork, the parameters of the hypernetwork are continuously optimized. When the parameters of the hypernetwork are adjusted to the optimal, the effect of searching for the self-attention mechanism representation network in the self-attention mechanism representation network search space through the hypernetwork with the optimal parameters is also optimal. The dataset for the semantic recognition task is input into the hypernetwork for the semantic recognition task. The hypernetwork for the semantic recognition task searches for the corresponding self-attention mechanism representation network from the self-attention mechanism representation network search space. This includes: the hypernetwork calculates the dataset for the semantic recognition task to obtain the top N self-attention mechanism representation networks with the highest scores, and determines the self-attention mechanism representation network with the highest score as the most suitable self-attention mechanism representation network for processing the dataset for the semantic recognition task.
2. The method according to claim 1, characterized in that, The search space for self-attention mechanism representation networks is constructed based on various architectures, including: Based on the structural features of the self-attention mechanism characterization network of the various architectures, multiple network fusion layers corresponding to each self-attention mechanism characterization network are obtained; The multiple network fusion layers are connected to obtain the multiple sub-networks; The self-attention mechanism is constructed using the multiple sub-networks to characterize the network search space.
3. The method according to claim 2, characterized in that, The method further includes: The self-attention mechanism is used to prune the network search space.
4. The method according to claim 3, characterized in that, The constraint strategies include: trunk constraint strategy, key-value binding constraint strategy, and zero-operation constraint strategy.
5. The method according to claim 1, characterized in that, The method further includes: The corresponding self-attention mechanism representation network is trained to obtain a trained self-attention mechanism representation network.
6. The method according to claim 2, characterized in that, Based on the structural features of the self-attention mechanism characterization networks of the various architectures, multiple network fusion layers corresponding to each self-attention mechanism characterization network are obtained, including: Based on the self-attention mechanism representation network of the various architectures, the self-attention mechanism fusion layer corresponding to each self-attention mechanism representation network is obtained; Based on the layers other than the self-attention layer in the self-attention mechanism representation network of the various architectures, an additive fusion layer corresponding to each self-attention mechanism representation network is obtained.
7. A self-attention mechanism characterizing network search device, characterized in that, The device includes: The search space construction module is used to construct a self-attention mechanism representation network search space based on self-attention mechanism representation networks of various architectures, including: constructing a self-attention mechanism representation network search space based on the structural characteristics of the self-attention mechanism representation networks of various architectures, wherein the self-attention mechanism representation network search space contains multiple sub-networks; A hypernetwork construction module is used to construct a hypernetwork from the multiple subnetworks. Specifically, parameters with the same context in the multiple subnetworks are set as parameters at the same position in the hypernetwork; parameters with different contexts in the multiple subnetworks are set as parameters at different positions in the hypernetwork. Specifically, training data for the semantic recognition task is collected, and the training data is put into a set to obtain the training dataset corresponding to the semantic recognition task. The training dataset corresponding to the semantic recognition task is input into the hypernetwork to train the hypernetwork and obtain a trained hypernetwork for the semantic recognition task. During the training process of the hypernetwork, the parameters of the hypernetwork are continuously optimized. When the parameters of the hypernetwork are adjusted to the optimal, the effect of searching for the self-attention mechanism representation network in the self-attention mechanism representation network search space through the hypernetwork with the optimal parameters also reaches the optimal. The network acquisition module is used to input a dataset for a semantic recognition task into the hypernetwork for the semantic recognition task, and search for the corresponding self-attention mechanism representation network from the self-attention mechanism representation network search space through the hypernetwork for the semantic recognition task. This includes: the hypernetwork calculates the dataset for the semantic recognition task to obtain the top N self-attention mechanism representation networks with the highest scores, and determines the self-attention mechanism representation network with the highest score as the most suitable self-attention mechanism representation network for processing the dataset for the semantic recognition task.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.